Nov 02 02:46:55.809649 ip-10-0-137-74 systemd[1]: Starting Kubernetes Kubelet... Nov 02 02:46:56.196437 ip-10-0-137-74 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 02:46:56.196437 ip-10-0-137-74 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 02:46:56.196437 ip-10-0-137-74 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 02:46:56.196437 ip-10-0-137-74 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 02:46:56.203261 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203191 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.203261 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203257 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.203261 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203261 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.203261 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203264 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203268 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203271 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203273 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203276 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203279 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203282 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203284 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203287 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203290 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203292 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203295 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203297 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203300 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203303 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203305 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203308 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203310 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203313 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203315 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.203392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203329 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203332 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203335 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203338 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203340 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203343 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203345 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203359 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203362 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203364 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203367 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203370 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203374 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203377 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203380 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203383 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203385 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203388 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203391 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203393 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.203857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203396 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203399 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203403 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203408 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203411 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203415 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203417 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203420 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203423 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203426 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203429 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203432 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203434 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203437 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203440 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203442 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203445 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203447 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203450 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.204315 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203452 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203456 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203459 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203461 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203464 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203467 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203469 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203473 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203475 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203478 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203481 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203485 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203487 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203490 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203492 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203495 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203498 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203500 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203503 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.204781 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203505 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203508 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203511 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203513 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203516 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203519 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203522 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203525 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203527 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203530 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203532 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203537 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203540 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203543 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203546 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203549 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203552 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203554 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203557 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.205232 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203560 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203563 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203566 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203568 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203571 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.203574 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205315 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205337 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205341 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205345 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205362 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205369 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205375 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205380 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205384 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205387 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205389 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205392 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205395 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.205723 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205398 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205401 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205404 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205406 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205409 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205411 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205414 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205416 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205419 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205422 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205424 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205427 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205429 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205432 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205434 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205438 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205441 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205444 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205447 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205450 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.206192 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205453 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205457 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205461 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205464 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205467 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205470 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205473 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205476 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205479 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205481 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205484 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205487 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205489 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205492 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205495 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205498 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205500 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205503 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205506 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.206750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205508 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205511 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205513 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205522 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205525 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205527 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205530 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205533 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205536 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205539 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205542 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205544 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205547 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205550 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205552 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205555 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205557 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205560 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205562 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205565 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.207206 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205567 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205570 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205572 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205575 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205577 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205580 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205582 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205586 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205588 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205591 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205593 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205596 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205599 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205601 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205604 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205607 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205609 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205612 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205615 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205617 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.207688 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205620 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205623 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205625 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205628 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205631 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205634 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205636 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205638 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205641 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205644 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205646 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205650 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205652 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.205655 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206114 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206119 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206122 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206124 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206127 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.208160 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206130 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206133 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206136 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206139 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206142 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206145 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206148 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206150 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206153 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206156 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206158 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206161 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206164 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206168 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206172 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206175 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206178 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206189 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206192 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.208620 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206196 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206198 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206201 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206204 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206207 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206210 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206212 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206215 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206218 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206220 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206223 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206225 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206228 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206231 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206234 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206236 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206239 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206242 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206245 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206248 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.209061 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206250 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206253 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206255 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206258 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206261 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206264 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206266 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206269 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206271 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206274 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206277 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206279 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206282 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206284 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206287 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206290 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206292 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206296 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206299 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.209532 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206303 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206306 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206308 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206311 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206313 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206316 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206319 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206321 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206325 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206327 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206330 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206333 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206335 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206338 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206340 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206343 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206346 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206363 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206367 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206369 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.209986 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206372 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206374 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206377 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206379 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206382 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206385 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206387 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206390 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206392 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206395 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206397 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206400 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206402 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206405 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206409 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206412 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206415 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206418 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206421 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206423 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.210507 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206426 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206429 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.206431 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207507 2584 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207516 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207523 2584 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207528 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207532 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207536 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207541 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207545 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207548 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207551 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207555 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207558 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207561 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207564 2584 flags.go:64] FLAG: --cgroup-root="" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207567 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207570 2584 flags.go:64] FLAG: --client-ca-file="" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207573 2584 flags.go:64] FLAG: --cloud-provider="external" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207576 2584 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207580 2584 flags.go:64] FLAG: --cluster-domain="" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207583 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207586 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 02:46:56.210971 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207590 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207593 2584 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207598 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207600 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207603 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207607 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207610 2584 flags.go:64] FLAG: --contention-profiling="false" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207613 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207616 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207619 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207622 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207627 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207630 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207632 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207635 2584 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207638 2584 flags.go:64] FLAG: --enable-server="true" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207641 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207646 2584 flags.go:64] FLAG: --event-burst="100" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207649 2584 flags.go:64] FLAG: --event-qps="50" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207652 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207655 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207658 2584 flags.go:64] FLAG: --eviction-hard="" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207662 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207665 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207668 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 02:46:56.211536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207671 2584 flags.go:64] FLAG: --eviction-soft="" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207674 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207677 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207680 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207683 2584 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207686 2584 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207689 2584 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207692 2584 flags.go:64] FLAG: --feature-gates="" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207696 2584 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207699 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207703 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207706 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207710 2584 flags.go:64] FLAG: --healthz-port="10248" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207713 2584 flags.go:64] FLAG: --help="false" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207715 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207719 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207722 2584 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207725 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207730 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207733 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207736 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207739 2584 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207742 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 02:46:56.212144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207745 2584 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207748 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207751 2584 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207754 2584 flags.go:64] FLAG: --kube-reserved="" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207757 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207760 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207763 2584 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207765 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207768 2584 flags.go:64] FLAG: --lock-file="" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207771 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207774 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207777 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207783 2584 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207785 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207789 2584 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207792 2584 flags.go:64] FLAG: --logging-format="text" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207794 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207798 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207801 2584 flags.go:64] FLAG: --manifest-url="" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207804 2584 flags.go:64] FLAG: --manifest-url-header="" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207808 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207811 2584 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207815 2584 flags.go:64] FLAG: --max-pods="110" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207818 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207821 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 02:46:56.212702 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207824 2584 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207827 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207830 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207834 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207837 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207845 2584 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207848 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207851 2584 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207854 2584 flags.go:64] FLAG: --pod-cidr="" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207857 2584 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207860 2584 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207863 2584 flags.go:64] FLAG: --pods-per-core="0" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207866 2584 flags.go:64] FLAG: --port="10250" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207869 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207872 2584 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ca8720f3b8cbc55d" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207875 2584 flags.go:64] FLAG: --qos-reserved="" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207878 2584 flags.go:64] FLAG: --read-only-port="10255" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207882 2584 flags.go:64] FLAG: --register-node="true" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207885 2584 flags.go:64] FLAG: --register-with-taints="" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207888 2584 flags.go:64] FLAG: --registry-burst="10" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207891 2584 flags.go:64] FLAG: --registry-qps="5" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207894 2584 flags.go:64] FLAG: --reserved-cpus="" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207898 2584 flags.go:64] FLAG: --reserved-memory="" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207901 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207904 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207907 2584 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 02:46:56.213284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207911 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207913 2584 flags.go:64] FLAG: --runonce="false" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207916 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207919 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207922 2584 flags.go:64] FLAG: --seccomp-default="false" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207925 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207928 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207931 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207934 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207937 2584 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207943 2584 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207946 2584 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207949 2584 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207951 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207955 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207958 2584 flags.go:64] FLAG: --system-cgroups="" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207960 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207966 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207969 2584 flags.go:64] FLAG: --tls-cert-file="" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207971 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207975 2584 flags.go:64] FLAG: --tls-min-version="" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207977 2584 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207980 2584 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207983 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207986 2584 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 02:46:56.214033 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207989 2584 flags.go:64] FLAG: --v="2" Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207994 2584 flags.go:64] FLAG: --version="false" Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.207998 2584 flags.go:64] FLAG: --vmodule="" Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.208002 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.208005 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208141 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208146 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208149 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208152 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208155 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208157 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208160 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208163 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208166 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208169 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208172 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208175 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208179 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208182 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208185 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.215088 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208188 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208191 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208193 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208196 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208198 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208201 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208203 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208206 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208209 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208211 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208214 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208216 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208219 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208222 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208225 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208227 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208230 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208232 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208236 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.216058 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208239 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208241 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208244 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208246 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208249 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208252 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208254 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208256 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208260 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208264 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208269 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208272 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208275 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208278 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208281 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208283 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208286 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208288 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208291 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208294 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.216873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208297 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208299 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208302 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208304 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208307 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208309 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208312 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208315 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208317 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208320 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208322 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208326 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208329 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208331 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208334 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208337 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208339 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208342 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208345 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208347 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.217739 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208360 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208363 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208367 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208370 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208373 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208377 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208380 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208383 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208386 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208388 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208391 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208394 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208396 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208399 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208401 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208404 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208407 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208409 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208412 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208415 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.218644 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208417 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208420 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208422 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208426 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208428 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208431 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208434 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208436 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208439 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208442 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208444 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208447 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.208473 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208516 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208520 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.219332 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208525 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208528 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208531 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208534 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208536 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208539 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208542 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208545 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208547 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208550 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208552 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208555 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208558 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208560 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208563 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208565 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208568 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208571 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208573 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.219857 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208576 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208580 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208583 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208586 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208589 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208592 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208594 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208597 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208599 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208602 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208604 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208607 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208609 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208614 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208617 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208621 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208624 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208627 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208630 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.220643 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208633 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208636 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208639 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208641 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208644 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208646 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208649 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208651 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208654 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208657 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208659 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208662 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208664 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208667 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208671 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208674 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208677 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208679 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208682 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208684 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.221440 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208687 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208690 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208692 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208695 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208697 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208700 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208704 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208706 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208709 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208711 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208714 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208720 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208724 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208727 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208730 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208733 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208736 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208739 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208742 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.222022 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208744 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208747 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208749 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208752 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208754 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208757 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208760 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208763 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208766 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208768 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208771 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208773 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208776 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208779 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208781 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208784 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208786 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208789 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208792 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208796 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.222806 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208798 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208801 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208804 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208807 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208809 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208813 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.208816 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.216773 2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.216790 2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.216811 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.216819 2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216924 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216933 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216939 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216943 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216947 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216952 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216957 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216961 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.223288 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216965 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216970 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216974 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216978 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216982 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216987 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.216992 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217004 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217009 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217013 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217017 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217021 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217025 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217030 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217034 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217038 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217043 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217047 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217052 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.223818 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217056 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217061 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217065 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217069 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217074 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217078 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217082 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217086 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217091 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217095 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217099 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217103 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217107 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217111 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217116 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217120 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217124 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217129 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217136 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217143 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.224257 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217148 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217160 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217165 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217169 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217174 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217178 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217182 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217187 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217191 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217195 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217200 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217218 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217222 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217226 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217230 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217235 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217239 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217245 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217252 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217257 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.224750 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217262 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217266 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217270 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217274 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217278 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217283 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217287 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217291 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217295 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217299 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217304 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217308 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217312 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217316 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217329 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217333 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217338 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217342 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217346 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217368 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.225220 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217372 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217377 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217381 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217386 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217390 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217394 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217398 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217402 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217407 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217411 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217415 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217419 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217423 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217427 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217431 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217435 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217440 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217444 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.225709 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217448 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.217490 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217792 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217801 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217805 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217809 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217814 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217819 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217824 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217835 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217840 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217844 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217848 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217853 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217857 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217862 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.226136 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217866 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217870 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217874 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217878 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217882 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217886 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217891 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217895 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217899 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217903 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217907 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217912 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217916 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217921 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217925 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217929 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217934 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217938 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217942 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.226536 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217947 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217951 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217955 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217959 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217964 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217969 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217973 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217984 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217991 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.217999 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218004 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218008 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218012 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218016 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218020 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218024 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218029 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218033 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218037 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218042 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.226981 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218046 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218050 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218054 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218058 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218062 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218066 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218071 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218075 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218079 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218084 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218088 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218092 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218096 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218101 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218105 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218109 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218113 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218118 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218122 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.227513 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218126 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218137 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218142 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218146 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218150 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218154 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218158 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218162 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218166 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218171 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218175 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218179 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218193 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218197 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218201 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218205 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218209 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218214 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218218 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.227951 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218222 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218226 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218230 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218234 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218239 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218243 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218247 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218253 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218259 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218264 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218268 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218272 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218276 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218280 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218284 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.218330 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 02:46:56.228487 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218412 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218422 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218427 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218432 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218436 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218440 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218444 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218449 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218453 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218457 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218461 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218465 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218469 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218474 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218478 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218482 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218486 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218490 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218495 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218499 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:46:56.228873 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218503 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218507 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218512 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218516 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218520 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218523 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218528 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218532 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218536 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218541 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218545 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218549 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218563 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218568 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218573 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218577 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218581 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218585 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218590 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:46:56.229361 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218594 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218598 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218602 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218606 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218610 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218614 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218619 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218623 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218627 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218633 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218638 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218643 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218647 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218651 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218655 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218659 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218663 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218667 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218671 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218675 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:46:56.229865 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218679 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218684 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218688 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218692 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218696 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218700 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218712 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218717 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218721 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218733 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218737 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218741 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218745 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218749 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218753 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218757 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218762 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218766 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218770 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218774 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:46:56.230337 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218778 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218782 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218787 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218791 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218795 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218799 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218803 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218808 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218814 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218818 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218822 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218826 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218830 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218834 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218839 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218843 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218847 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218852 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218858 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:46:56.230853 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218872 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218877 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218883 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218887 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218891 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218896 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218900 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:56.218904 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.218987 2584 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.222309 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.223624 2584 server.go:1014] "Starting client certificate rotation" Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.223720 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.223769 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.225687 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 02:46:56.231308 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.228381 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 02:46:56.235931 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.235916 2584 log.go:25] "Validated CRI v1 runtime API" Nov 02 02:46:56.239681 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.239665 2584 log.go:25] "Validated CRI v1 image API" Nov 02 02:46:56.240548 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.240523 2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 02:46:56.245205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.245186 2584 fs.go:135] Filesystem UUIDs: map[00a52bc1-5abc-4a8b-a2e6-06c32f0beb90:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe09c98c-e710-4552-8994-e21049e9cc1c:/dev/nvme0n1p3] Nov 02 02:46:56.245255 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.245205 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 02:46:56.250316 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.250204 2584 manager.go:217] Machine: {Timestamp:2026-11-02 02:46:56.248788648 +0000 UTC m=+0.340391637 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103732 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d28ca222015037014e343675fb299 SystemUUID:ec2d28ca-2220-1503-7014-e343675fb299 BootID:d41b5709-d316-4fe1-b0b1-462a60c78450 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:ef:3c:01:63 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:ef:3c:01:63 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:e0:a2:15:cd:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 02:46:56.250316 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.250316 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 02:46:56.250433 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.250421 2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 02:46:56.254081 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.254057 2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 02:46:56.254214 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.254084 2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-74.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 02:46:56.254257 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.254217 2584 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 02:46:56.254257 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.254225 2584 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 02:46:56.254257 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.254251 2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 02:46:56.254328 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.254265 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 02:46:56.255547 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.255538 2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 02:46:56.260046 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260034 2584 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 02:46:56.260165 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260157 2584 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 02:46:56.260961 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260952 2584 kubelet.go:493] "Attempting to sync node with API server" Nov 02 02:46:56.260992 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260965 2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 02:46:56.260992 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260974 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 02:46:56.260992 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260984 2584 kubelet.go:405] "Adding apiserver pod source" Nov 02 02:46:56.260992 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.260992 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 02:46:56.261376 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.261362 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 02:46:56.269999 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.269959 2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 02:46:56.276918 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.276902 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 02:46:56.277799 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277787 2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 02:46:56.277851 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277823 2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 02:46:56.277968 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277958 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 02:46:56.278014 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277974 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 02:46:56.278014 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277980 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 02:46:56.278014 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277985 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 02:46:56.278014 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277991 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 02:46:56.278014 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.277996 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 02:46:56.278732 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.278723 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 02:46:56.278762 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.278734 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 02:46:56.278762 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.278741 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 02:46:56.278762 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.278746 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 02:46:56.278762 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.278754 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 02:46:56.278762 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.278762 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 02:46:56.279562 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.279553 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 02:46:56.279592 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.279563 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 02:46:56.283238 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.283225 2584 server.go:1263] "Started kubelet" Nov 02 02:46:56.283379 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.283317 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 02:46:56.283438 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.283396 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 02:46:56.283438 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.283392 2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 02:46:56.283896 ip-10-0-137-74 systemd[1]: Started Kubernetes Kubelet. Nov 02 02:46:56.284942 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.284923 2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 02:46:56.285223 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.285208 2584 server.go:317] "Adding debug handlers to kubelet server" Nov 02 02:46:56.290099 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.290070 2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 02:46:56.292930 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.292913 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 02:46:56.294285 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.294268 2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 02:46:56.294515 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.294488 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.295088 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295072 2584 factory.go:55] Registering systemd factory Nov 02 02:46:56.295088 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295091 2584 factory.go:223] Registration of the systemd container factory successfully Nov 02 02:46:56.295265 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295248 2584 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 02:46:56.295265 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295266 2584 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 02:46:56.295432 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295289 2584 factory.go:153] Registering CRI-O factory Nov 02 02:46:56.295432 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295248 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 02:46:56.295432 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295308 2584 factory.go:223] Registration of the crio container factory successfully Nov 02 02:46:56.295432 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295383 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 02:46:56.295432 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295402 2584 factory.go:103] Registering Raw factory Nov 02 02:46:56.295432 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295416 2584 manager.go:1199] Started watching for new ooms in manager Nov 02 02:46:56.295614 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295385 2584 reconstruct.go:97] "Volume reconstruction finished" Nov 02 02:46:56.295614 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.295458 2584 reconciler.go:29] "Reconciler: start to sync state" Nov 02 02:46:56.296433 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.296419 2584 manager.go:319] Starting recovery of all containers Nov 02 02:46:56.306972 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.306951 2584 manager.go:324] Recovery completed Nov 02 02:46:56.311650 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.311635 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.313954 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.313938 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.314053 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.313968 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.314053 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.313982 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.314729 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.314712 2584 cpu_manager.go:226] "Starting" policy="none" Nov 02 02:46:56.314729 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.314728 2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 02:46:56.314829 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.314748 2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 02:46:56.314829 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.314718 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.315444 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.315428 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.315517 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.315456 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.315517 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.315466 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.317027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.317013 2584 policy_none.go:50] "Start" Nov 02 02:46:56.317027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.317029 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 02:46:56.317105 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.317040 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 02:46:56.395505 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.395485 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.421765 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.421743 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-74.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 02:46:56.421765 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.421745 2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-74.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 02:46:56.422847 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.422026 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-74.us-west-1.compute.internal.18e41a6f26198c8c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-74.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-74.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 02:46:56.283200652 +0000 UTC m=+0.374803639,LastTimestamp:2026-11-02 02:46:56.283200652 +0000 UTC m=+0.374803639,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-74.us-west-1.compute.internal,}" Nov 02 02:46:56.464239 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.464176 2584 policy_none.go:44] "Start" Nov 02 02:46:56.491612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.475580 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 02:46:56.491612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.475804 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-74.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 02:46:56.491612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.475817 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 02:46:56.491612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.475893 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-74.us-west-1.compute.internal.18e41a6f27eecdd2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-74.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-74.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-74.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 02:46:56.313953746 +0000 UTC m=+0.405556733,LastTimestamp:2026-11-02 02:46:56.313953746 +0000 UTC m=+0.405556733,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-74.us-west-1.compute.internal,}" Nov 02 02:46:56.493973 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.493951 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 02:46:56.494036 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.493992 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9ktjf" Nov 02 02:46:56.495953 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.495937 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.499895 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.499871 2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 02:46:56.502036 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502024 2584 manager.go:341] "Starting Device Plugin manager" Nov 02 02:46:56.502476 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.502462 2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 02:46:56.502535 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502479 2584 server.go:87] "Starting device plugin registration server" Nov 02 02:46:56.502730 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502719 2584 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 02:46:56.502762 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502733 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 02:46:56.502849 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502828 2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 02:46:56.502955 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502904 2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 02:46:56.502955 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.502914 2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 02:46:56.503365 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.503331 2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 02:46:56.503437 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.503390 2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.511418 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.511396 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 02:46:56.512558 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.512539 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 02:46:56.512558 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.512559 2584 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 02:46:56.512665 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.512574 2584 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 02:46:56.512665 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.512606 2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 02:46:56.577964 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.577948 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:46:56.603169 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.603156 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.603959 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.603939 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.604037 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.603972 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.604037 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.603984 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.604037 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.604006 2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.613203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.613187 2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal"] Nov 02 02:46:56.613247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.613239 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.613944 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.613929 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.613991 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.613958 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.613991 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.613969 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.616199 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.616178 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.616665 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.616649 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.616713 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.616678 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.617538 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.617520 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.617609 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.617544 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.617609 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.617554 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.617609 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.617524 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.617696 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.617624 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.617696 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.617643 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.619748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.619731 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.619834 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.619756 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:56.620525 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.620511 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:56.620606 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.620538 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:56.620606 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.620551 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:56.645989 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.645973 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.648559 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.648544 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.660746 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.660726 2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.660746 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.660745 2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-74.us-west-1.compute.internal\": node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.696359 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.696338 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f403bae9d2c2c1c33d0488e61f8d37c6-config\") pod \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"f403bae9d2c2c1c33d0488e61f8d37c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.767462 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.767415 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.796723 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.796705 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f403bae9d2c2c1c33d0488e61f8d37c6-config\") pod \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"f403bae9d2c2c1c33d0488e61f8d37c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.796780 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.796732 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.796780 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.796751 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.796856 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.796795 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f403bae9d2c2c1c33d0488e61f8d37c6-config\") pod \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"f403bae9d2c2c1c33d0488e61f8d37c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.868202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.868172 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:56.897489 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.897473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.897546 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.897495 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.897546 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.897519 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.897616 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.897552 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.948481 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.948464 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.951391 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:56.951377 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:56.968497 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:56.968376 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:57.069090 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:57.069044 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:57.169119 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:57.169092 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:57.224119 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.224101 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 02:46:57.224489 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.224195 2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 02:46:57.224489 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:57.224270 2584 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af3a1c6fd9c4d43e7823e9d8be5f460a-a3480c90fef6e68b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-74.us-west-1.compute.internal.18e41a6f27ef1f41\": read tcp 10.0.137.74:57348->3.151.167.219:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-74.us-west-1.compute.internal.18e41a6f27ef1f41 default 4842 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-74.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-74.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-74.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 02:46:56 +0000 UTC,LastTimestamp:2026-11-02 02:46:56.613963104 +0000 UTC m=+0.705566093,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-74.us-west-1.compute.internal,}" Nov 02 02:46:57.269631 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:57.269609 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:57.369761 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:57.369721 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:57.470740 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:57.470720 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:46:57.476829 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.476811 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:46:57.477720 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.477707 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:46:57.477785 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.477732 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:46:57.477785 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.477742 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:46:57.492461 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.492448 2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:46:57.495497 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.495483 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:57.498615 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.498594 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 02:41:56 +0000 UTC" deadline="2028-05-01 06:35:19.012386471 +0000 UTC" Nov 02 02:46:57.498663 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.498616 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13107h48m21.513773157s" Nov 02 02:46:57.555731 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.555713 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 02:46:57.557247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.557235 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:46:57.609534 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.609514 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 02:46:57.696231 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.696188 2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:46:57.979687 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:57.979629 2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:46:58.023769 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.023731 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60aa7d07786267e9b48b181b1bcbc45b.slice/crio-b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca WatchSource:0}: Error finding container b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca: Status 404 returned error can't find the container with id b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca Nov 02 02:46:58.024111 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.024087 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf403bae9d2c2c1c33d0488e61f8d37c6.slice/crio-9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f WatchSource:0}: Error finding container 9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f: Status 404 returned error can't find the container with id 9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f Nov 02 02:46:58.028200 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.028188 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 02:46:58.262328 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.262282 2584 apiserver.go:52] "Watching apiserver" Nov 02 02:46:58.294057 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.294040 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 02:46:58.355781 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.355761 2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 02:46:58.356027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.356011 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-cvzpz","kube-system/global-pull-secret-syncer-xhmws","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l","openshift-network-diagnostics/network-check-target-d42js","kube-system/konnectivity-agent-v8xt2","kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-76cvn","openshift-dns/node-resolver-rfzxg","openshift-multus/multus-additional-cni-plugins-n56m9","openshift-image-registry/node-ca-5sb5c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal","openshift-multus/multus-9kpxq","openshift-network-operator/iptables-alerter-zmp76","openshift-ovn-kubernetes/ovnkube-node-q6wpj"] Nov 02 02:46:58.359299 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.359284 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.361183 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.361164 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.363023 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.363008 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.364920 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.364904 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:46:58.365004 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.364952 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:46:58.366828 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.366814 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.369568 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.369441 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.371637 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.371620 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.373616 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.373599 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:58.373684 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.373662 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:46:58.375616 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.375597 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.375674 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.375659 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:46:58.377621 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.377606 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.379966 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.379953 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.382032 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.382017 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.398159 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.396545 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 02:46:58.404902 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.404885 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.404988 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.404907 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-cni-binary-copy\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.404988 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.404921 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-netns\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.404988 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.404934 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysconfig\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.404988 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.404952 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41d1c58c-4c19-4212-934c-ecaf59792f37-host-slash\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.405129 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405034 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.405129 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405076 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-systemd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405129 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405098 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-k8s-cni-cncf-io\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.405129 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405116 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-systemd\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405241 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405154 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-sys\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405241 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405174 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-sys-fs\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.405241 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405205 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-bin\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405241 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405218 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-socket-dir-parent\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.405241 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405231 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-lib-modules\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405243 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-tmp\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405272 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bd78\" (UniqueName: \"kubernetes.io/projected/9ba41133-2ab9-43ac-94d9-adfade07fee6-kube-api-access-5bd78\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405300 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-system-cni-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405316 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grznx\" (UniqueName: \"kubernetes.io/projected/b855c6cb-d565-4687-bd13-248d85111459-kube-api-access-grznx\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-konnectivity-ca\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405374 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-config\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405391 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-tmp-dir\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405403 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405419 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405439 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67nx\" (UniqueName: \"kubernetes.io/projected/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-kube-api-access-d67nx\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405460 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-multus-certs\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405474 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-kubelet\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405487 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-ovn\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405500 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405546 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-kubernetes\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405576 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-registration-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405601 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405631 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-var-lib-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405659 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-netd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405677 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnsj\" (UniqueName: \"kubernetes.io/projected/41d1c58c-4c19-4212-934c-ecaf59792f37-kube-api-access-6hnsj\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405693 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jb6m\" (UniqueName: \"kubernetes.io/projected/60e09d2f-2971-476b-afd8-b3b37be1437b-kube-api-access-6jb6m\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405710 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-kubelet-config\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.405748 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405732 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405767 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-node-log\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405819 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-system-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405853 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-var-lib-kubelet\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405871 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-tuned\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405893 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405914 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-netns\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405927 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91547ba7-4770-461b-a34b-0f3a46bdde43-host\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405939 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-conf-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405964 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.405997 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406020 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgmn\" (UniqueName: \"kubernetes.io/projected/17e048e5-425c-43fe-8453-f4e30a51df0a-kube-api-access-9kgmn\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406046 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-etc-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406060 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-env-overrides\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406089 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-bin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406138 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-script-lib\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406166 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406168 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf47p\" (UniqueName: \"kubernetes.io/projected/42726530-4a21-425e-8598-05227546e93f-kube-api-access-bf47p\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406189 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-run\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406211 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-host\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406233 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-socket-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406259 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-cnibin\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406295 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-dbus\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406333 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-systemd-units\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406387 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-log-socket\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406410 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-hosts-file\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406435 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406456 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91547ba7-4770-461b-a34b-0f3a46bdde43-serviceca\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406472 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-etc-kubernetes\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406493 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-modprobe-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406524 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-conf\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406542 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41d1c58c-4c19-4212-934c-ecaf59792f37-iptables-alerter-script\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406555 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-device-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.406778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406576 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-multus\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406619 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-kubelet\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406640 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406653 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovn-node-metrics-cert\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406683 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhhr\" (UniqueName: \"kubernetes.io/projected/91547ba7-4770-461b-a34b-0f3a46bdde43-kube-api-access-xvhhr\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406698 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-cnibin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406733 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406766 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-slash\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406807 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-hostroot\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406833 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-multus-daemon-config\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406861 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406901 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-agent-certs\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.407211 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.406925 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz78w\" (UniqueName: \"kubernetes.io/projected/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-kube-api-access-xz78w\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.410739 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410722 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 02:46:58.410850 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410837 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b8q6l\"" Nov 02 02:46:58.410910 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410850 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.410910 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410897 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ghkrt\"" Nov 02 02:46:58.411001 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410926 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.411001 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410930 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 02:46:58.411001 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410929 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.411001 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410961 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.411001 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410983 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.411001 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.410984 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.411179 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.411008 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cgnbb\"" Nov 02 02:46:58.413690 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.413677 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 02:46:58.413690 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.413688 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 02:46:58.413772 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.413688 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nvmsr\"" Nov 02 02:46:58.416061 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.416049 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 02:46:58.416116 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.416074 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.501182 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.501163 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 02:46:58.501263 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.501193 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l67zs" Nov 02 02:46:58.505486 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505472 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 02:46:58.505580 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505561 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.505580 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505577 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.505699 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505646 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 02:46:58.505699 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505674 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505908 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505919 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505930 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505940 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505932 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pbbl9\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505914 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505916 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505948 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 02:46:58.505979 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.505968 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7jkxs\"" Nov 02 02:46:58.506277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.506000 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 02:46:58.506277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.506001 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5mjxc\"" Nov 02 02:46:58.506277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.506124 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5v9mr\"" Nov 02 02:46:58.506277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.506177 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 02:46:58.506277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.506213 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cjmh9\"" Nov 02 02:46:58.506277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.506216 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 02:46:58.507310 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507297 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-netd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507361 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507317 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnsj\" (UniqueName: \"kubernetes.io/projected/41d1c58c-4c19-4212-934c-ecaf59792f37-kube-api-access-6hnsj\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.507361 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507332 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jb6m\" (UniqueName: \"kubernetes.io/projected/60e09d2f-2971-476b-afd8-b3b37be1437b-kube-api-access-6jb6m\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.507361 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-kubelet-config\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.507478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507374 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507397 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-node-log\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507416 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-system-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.507478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507420 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-netd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507434 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507467 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-node-log\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507487 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-kubelet-config\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507509 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-var-lib-kubelet\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507517 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-system-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507534 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-tuned\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507555 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-var-lib-kubelet\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507560 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507608 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-netns\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507625 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91547ba7-4770-461b-a34b-0f3a46bdde43-host\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507648 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-netns\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507649 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91547ba7-4770-461b-a34b-0f3a46bdde43-host\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507652 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-conf-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507686 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-conf-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507703 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507722 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.507728 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507736 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgmn\" (UniqueName: \"kubernetes.io/projected/17e048e5-425c-43fe-8453-f4e30a51df0a-kube-api-access-9kgmn\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507749 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507751 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-etc-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507779 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-etc-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507783 2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507787 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-env-overrides\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507842 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-bin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507863 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-script-lib\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507874 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-bin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf47p\" (UniqueName: \"kubernetes.io/projected/42726530-4a21-425e-8598-05227546e93f-kube-api-access-bf47p\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507905 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-run\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507924 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-host\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.507960 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-host\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508044 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-run\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508056 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508071 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-socket-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508093 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-cnibin\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508114 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-dbus\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.508389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508134 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-systemd-units\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508156 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-log-socket\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508169 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-socket-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508179 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-systemd-units\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508187 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508178 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-hosts-file\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508209 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-env-overrides\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508156 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-cnibin\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508234 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-log-socket\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508232 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508248 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-hosts-file\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508254 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508265 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91547ba7-4770-461b-a34b-0f3a46bdde43-serviceca\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508279 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-dbus\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508285 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-etc-kubernetes\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508309 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-etc-kubernetes\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-modprobe-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.509024 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508337 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-conf\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508378 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41d1c58c-4c19-4212-934c-ecaf59792f37-iptables-alerter-script\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508400 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-device-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508420 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-multus\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508400 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-modprobe-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508442 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-kubelet\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508456 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-script-lib\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508468 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508459 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-conf\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508489 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-device-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovn-node-metrics-cert\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508504 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-kubelet\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508535 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-multus\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508539 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhhr\" (UniqueName: \"kubernetes.io/projected/91547ba7-4770-461b-a34b-0f3a46bdde43-kube-api-access-xvhhr\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508575 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-cnibin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.508591 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508597 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508621 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-slash\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.509672 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508629 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91547ba7-4770-461b-a34b-0f3a46bdde43-serviceca\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.508661 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret podName:f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c nodeName:}" failed. No retries permitted until 2026-11-02 02:46:59.008623046 +0000 UTC m=+3.100226037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret") pod "global-pull-secret-syncer-xhmws" (UID: "f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c") : object "kube-system"/"original-pull-secret" not registered Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508666 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-slash\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508695 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-hostroot\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508714 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-hostroot\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-multus-daemon-config\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508697 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-cnibin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.508735 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508741 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508762 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-agent-certs\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508783 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz78w\" (UniqueName: \"kubernetes.io/projected/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-kube-api-access-xz78w\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508805 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508834 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-cni-binary-copy\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508837 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41d1c58c-4c19-4212-934c-ecaf59792f37-iptables-alerter-script\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.508846 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:46:59.00883003 +0000 UTC m=+3.100433016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508877 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-netns\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508886 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.510457 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508879 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508911 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-netns\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.508913 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysconfig\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509003 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41d1c58c-4c19-4212-934c-ecaf59792f37-host-slash\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509071 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-systemd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509110 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysconfig\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509146 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-k8s-cni-cncf-io\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509164 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-systemd\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509169 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-k8s-cni-cncf-io\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509179 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-sys\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509211 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-systemd\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509204 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-sys-fs\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509237 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-bin\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509250 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-systemd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509255 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-socket-dir-parent\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509280 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-sys\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509285 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-lib-modules\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511203 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509309 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-tmp\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509313 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41d1c58c-4c19-4212-934c-ecaf59792f37-host-slash\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509321 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-cni-binary-copy\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509333 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-multus-daemon-config\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509340 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bd78\" (UniqueName: \"kubernetes.io/projected/9ba41133-2ab9-43ac-94d9-adfade07fee6-kube-api-access-5bd78\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509394 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-socket-dir-parent\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509410 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-system-cni-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509425 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-sys-fs\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509483 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grznx\" (UniqueName: \"kubernetes.io/projected/b855c6cb-d565-4687-bd13-248d85111459-kube-api-access-grznx\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509586 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-lib-modules\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509609 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-bin\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509632 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-system-cni-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509658 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509659 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-konnectivity-ca\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-config\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509821 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-tmp-dir\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509844 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.511842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509866 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509887 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67nx\" (UniqueName: \"kubernetes.io/projected/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-kube-api-access-d67nx\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509912 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-multus-certs\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509932 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-kubelet\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.509954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-ovn\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510006 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-ovn\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510246 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-multus-certs\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510266 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-konnectivity-ca\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510322 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-kubelet\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510366 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510381 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510390 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-kubernetes\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510407 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510412 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-registration-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510408 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-tmp-dir\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510441 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-kubernetes\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510455 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.512247 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510483 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-var-lib-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510503 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-registration-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.510533 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-var-lib-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.511335 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-config\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.511334 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.511596 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-tmp\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.511719 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-tuned\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.511824 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovn-node-metrics-cert\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.512807 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.511881 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-agent-certs\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.516020 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.515963 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"f403bae9d2c2c1c33d0488e61f8d37c6","Type":"ContainerStarted","Data":"9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f"} Nov 02 02:46:58.516885 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.516866 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"60aa7d07786267e9b48b181b1bcbc45b","Type":"ContainerStarted","Data":"b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca"} Nov 02 02:46:58.559320 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.559296 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 02:46:58.559666 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.559653 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 02:46:58.559760 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.559750 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k6dmj for pod openshift-network-diagnostics/network-check-target-d42js: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:46:58.559881 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:58.559866 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj podName:bb2c8c59-10f2-4f93-a4bd-1b37406fec26 nodeName:}" failed. No retries permitted until 2026-11-02 02:46:59.059847441 +0000 UTC m=+3.151450430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k6dmj" (UniqueName: "kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj") pod "network-check-target-d42js" (UID: "bb2c8c59-10f2-4f93-a4bd-1b37406fec26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:46:58.559993 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.559901 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnsj\" (UniqueName: \"kubernetes.io/projected/41d1c58c-4c19-4212-934c-ecaf59792f37-kube-api-access-6hnsj\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.560072 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.559763 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf47p\" (UniqueName: \"kubernetes.io/projected/42726530-4a21-425e-8598-05227546e93f-kube-api-access-bf47p\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.560072 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.559918 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jb6m\" (UniqueName: \"kubernetes.io/projected/60e09d2f-2971-476b-afd8-b3b37be1437b-kube-api-access-6jb6m\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.560148 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.560123 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgmn\" (UniqueName: \"kubernetes.io/projected/17e048e5-425c-43fe-8453-f4e30a51df0a-kube-api-access-9kgmn\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.560835 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.560816 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhhr\" (UniqueName: \"kubernetes.io/projected/91547ba7-4770-461b-a34b-0f3a46bdde43-kube-api-access-xvhhr\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.561385 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.561367 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bd78\" (UniqueName: \"kubernetes.io/projected/9ba41133-2ab9-43ac-94d9-adfade07fee6-kube-api-access-5bd78\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.561574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.561555 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz78w\" (UniqueName: \"kubernetes.io/projected/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-kube-api-access-xz78w\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.561646 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.561605 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grznx\" (UniqueName: \"kubernetes.io/projected/b855c6cb-d565-4687-bd13-248d85111459-kube-api-access-grznx\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:58.561838 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.561819 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67nx\" (UniqueName: \"kubernetes.io/projected/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-kube-api-access-d67nx\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.642854 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.642835 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:46:58.668961 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.668945 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:46:58.674751 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.674738 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:46:58.679215 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.679195 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba41133_2ab9_43ac_94d9_adfade07fee6.slice/crio-9d2b1ed9084d3c23424bcf7ae2eab2bcc11bdbb9b8fc94ad24a4156948d596ae WatchSource:0}: Error finding container 9d2b1ed9084d3c23424bcf7ae2eab2bcc11bdbb9b8fc94ad24a4156948d596ae: Status 404 returned error can't find the container with id 9d2b1ed9084d3c23424bcf7ae2eab2bcc11bdbb9b8fc94ad24a4156948d596ae Nov 02 02:46:58.683267 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.683246 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d1c58c_4c19_4212_934c_ecaf59792f37.slice/crio-4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6 WatchSource:0}: Error finding container 4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6: Status 404 returned error can't find the container with id 4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6 Nov 02 02:46:58.683861 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.683845 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:46:58.684959 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.684942 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:46:58.692682 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.692661 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e09d2f_2971_476b_afd8_b3b37be1437b.slice/crio-484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6 WatchSource:0}: Error finding container 484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6: Status 404 returned error can't find the container with id 484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6 Nov 02 02:46:58.694980 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.694965 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecd4ba6_46cc_4f76_8984_492c35ad4f40.slice/crio-e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d WatchSource:0}: Error finding container e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d: Status 404 returned error can't find the container with id e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d Nov 02 02:46:58.705803 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.705791 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:46:58.710997 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.710980 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:46:58.713436 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.713421 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91547ba7_4770_461b_a34b_0f3a46bdde43.slice/crio-e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8 WatchSource:0}: Error finding container e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8: Status 404 returned error can't find the container with id e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8 Nov 02 02:46:58.714534 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.714474 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9kpxq" Nov 02 02:46:58.718048 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.718027 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf0a3b_df4f_458d_b0ae_72483d95f2d7.slice/crio-f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834 WatchSource:0}: Error finding container f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834: Status 404 returned error can't find the container with id f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834 Nov 02 02:46:58.718522 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.718504 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:46:58.721949 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.721932 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42726530_4a21_425e_8598_05227546e93f.slice/crio-7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4 WatchSource:0}: Error finding container 7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4: Status 404 returned error can't find the container with id 7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4 Nov 02 02:46:58.723985 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:58.723969 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:46:58.726073 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.726055 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a3ee91_5a11_4a4d_8b6d_32273c3c26b5.slice/crio-ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07 WatchSource:0}: Error finding container ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07: Status 404 returned error can't find the container with id ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07 Nov 02 02:46:58.731508 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:46:58.731492 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e048e5_425c_43fe_8453_f4e30a51df0a.slice/crio-6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852 WatchSource:0}: Error finding container 6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852: Status 404 returned error can't find the container with id 6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852 Nov 02 02:46:59.013098 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.013071 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:46:59.013278 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.013112 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:46:59.013278 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.013214 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 02:46:59.013278 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.013219 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:46:59.013278 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.013272 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret podName:f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c nodeName:}" failed. No retries permitted until 2026-11-02 02:47:00.013255902 +0000 UTC m=+4.104858903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret") pod "global-pull-secret-syncer-xhmws" (UID: "f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c") : object "kube-system"/"original-pull-secret" not registered Nov 02 02:46:59.013463 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.013317 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:00.013283295 +0000 UTC m=+4.104886285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:46:59.114398 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.114369 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:46:59.114528 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.114507 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 02:46:59.114528 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.114522 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 02:46:59.114604 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.114532 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k6dmj for pod openshift-network-diagnostics/network-check-target-d42js: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:46:59.114604 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.114583 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj podName:bb2c8c59-10f2-4f93-a4bd-1b37406fec26 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:00.114568342 +0000 UTC m=+4.206171319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6dmj" (UniqueName: "kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj") pod "network-check-target-d42js" (UID: "bb2c8c59-10f2-4f93-a4bd-1b37406fec26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:46:59.503842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.503759 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 02:41:58 +0000 UTC" deadline="2028-06-07 16:28:05.330873787 +0000 UTC" Nov 02 02:46:59.503842 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.503791 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14005h41m5.827087286s" Nov 02 02:46:59.513055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.513031 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:46:59.513172 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:46:59.513132 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:46:59.520973 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.520909 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerStarted","Data":"6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852"} Nov 02 02:46:59.523106 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.523034 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07"} Nov 02 02:46:59.524383 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.524257 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9kpxq" event={"ID":"42726530-4a21-425e-8598-05227546e93f","Type":"ContainerStarted","Data":"7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4"} Nov 02 02:46:59.527027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.527009 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" event={"ID":"60e09d2f-2971-476b-afd8-b3b37be1437b","Type":"ContainerStarted","Data":"484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6"} Nov 02 02:46:59.529695 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.529672 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfzxg" event={"ID":"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7","Type":"ContainerStarted","Data":"f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834"} Nov 02 02:46:59.531125 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.531067 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5sb5c" event={"ID":"91547ba7-4770-461b-a34b-0f3a46bdde43","Type":"ContainerStarted","Data":"e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8"} Nov 02 02:46:59.532271 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.532252 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v8xt2" event={"ID":"3ecd4ba6-46cc-4f76-8984-492c35ad4f40","Type":"ContainerStarted","Data":"e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d"} Nov 02 02:46:59.534388 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.534368 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zmp76" event={"ID":"41d1c58c-4c19-4212-934c-ecaf59792f37","Type":"ContainerStarted","Data":"4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6"} Nov 02 02:46:59.535732 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:46:59.535711 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-76cvn" event={"ID":"9ba41133-2ab9-43ac-94d9-adfade07fee6","Type":"ContainerStarted","Data":"9d2b1ed9084d3c23424bcf7ae2eab2bcc11bdbb9b8fc94ad24a4156948d596ae"} Nov 02 02:47:00.022634 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.022599 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:00.022841 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.022654 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:00.022841 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.022805 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:00.022947 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.022863 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:02.022846756 +0000 UTC m=+6.114449743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:00.022947 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.022922 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:00.023047 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.022955 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret podName:f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c nodeName:}" failed. No retries permitted until 2026-11-02 02:47:02.022945599 +0000 UTC m=+6.114548592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret") pod "global-pull-secret-syncer-xhmws" (UID: "f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c") : object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:00.123155 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.123115 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:00.123317 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.123237 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 02:47:00.123317 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.123250 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 02:47:00.123317 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.123258 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k6dmj for pod openshift-network-diagnostics/network-check-target-d42js: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:00.123317 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.123297 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj podName:bb2c8c59-10f2-4f93-a4bd-1b37406fec26 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:02.123285573 +0000 UTC m=+6.214888559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6dmj" (UniqueName: "kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj") pod "network-check-target-d42js" (UID: "bb2c8c59-10f2-4f93-a4bd-1b37406fec26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:00.504056 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.503923 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 02:41:58 +0000 UTC" deadline="2028-07-22 20:39:42.266682512 +0000 UTC" Nov 02 02:47:00.504056 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.503980 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15089h52m41.762706604s" Nov 02 02:47:00.513960 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.513777 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:00.513960 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.513904 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:00.514506 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:00.514272 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:00.514506 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:00.514456 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:01.512945 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:01.512884 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:01.513519 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:01.513003 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:01.561374 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:01.560124 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"f403bae9d2c2c1c33d0488e61f8d37c6","Type":"ContainerStarted","Data":"fac56e3d1f49e2a1f216d4bf826db52ddfff4adb7cd8d780b93e140cadc778d1"} Nov 02 02:47:02.040683 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.040634 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:02.040683 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.040681 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:02.040922 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.040827 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:02.040922 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.040891 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:06.040873327 +0000 UTC m=+10.132476308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:02.041035 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.040952 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:02.041035 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.040983 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret podName:f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c nodeName:}" failed. No retries permitted until 2026-11-02 02:47:06.040972954 +0000 UTC m=+10.132575934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret") pod "global-pull-secret-syncer-xhmws" (UID: "f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c") : object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:02.141426 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.141388 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:02.141580 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.141570 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 02:47:02.141636 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.141592 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 02:47:02.141636 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.141604 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k6dmj for pod openshift-network-diagnostics/network-check-target-d42js: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:02.141752 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.141665 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj podName:bb2c8c59-10f2-4f93-a4bd-1b37406fec26 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:06.141639155 +0000 UTC m=+10.233242138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6dmj" (UniqueName: "kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj") pod "network-check-target-d42js" (UID: "bb2c8c59-10f2-4f93-a4bd-1b37406fec26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:02.519636 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.519610 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:02.520277 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.519742 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:02.520277 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.519779 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:02.520277 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:02.519873 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:02.567500 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.567427 2584 generic.go:356] "Generic (PLEG): container finished" podID="60aa7d07786267e9b48b181b1bcbc45b" containerID="cb74b861e3a9e8f7428631b6f2c7cf02041e66fbbc6d868e783b197576b45db2" exitCode=0 Nov 02 02:47:02.567650 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.567513 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"60aa7d07786267e9b48b181b1bcbc45b","Type":"ContainerDied","Data":"cb74b861e3a9e8f7428631b6f2c7cf02041e66fbbc6d868e783b197576b45db2"} Nov 02 02:47:02.572668 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.572464 2584 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="df0ce24cec1139d3599fe0d8a06b9bbd09880cd158d51ede8fb953538cd0d9c5" exitCode=0 Nov 02 02:47:02.572668 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.572587 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerDied","Data":"df0ce24cec1139d3599fe0d8a06b9bbd09880cd158d51ede8fb953538cd0d9c5"} Nov 02 02:47:02.686150 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:02.686081 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" podStartSLOduration=5.686060449 podStartE2EDuration="5.686060449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:01.676321568 +0000 UTC m=+5.767924566" watchObservedRunningTime="2026-11-02 02:47:02.686060449 +0000 UTC m=+6.777663448" Nov 02 02:47:03.514445 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:03.514367 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:03.514606 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:03.514509 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:03.580823 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:03.580785 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v8xt2" event={"ID":"3ecd4ba6-46cc-4f76-8984-492c35ad4f40","Type":"ContainerStarted","Data":"b08d24345680e35627234b3d46aa9f67300fd8fe1e9ecc2d46f998d419af8e47"} Nov 02 02:47:03.591197 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:03.591169 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"60aa7d07786267e9b48b181b1bcbc45b","Type":"ContainerStarted","Data":"4cbf679b5057c4e04095965b6ec130bd5c1d16242bb75e0c78f324cabadf83c0"} Nov 02 02:47:03.807949 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:03.807833 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" podStartSLOduration=6.807812726 podStartE2EDuration="6.807812726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:03.807412209 +0000 UTC m=+7.899015256" watchObservedRunningTime="2026-11-02 02:47:03.807812726 +0000 UTC m=+7.899415726" Nov 02 02:47:03.808597 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:03.808550 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v8xt2" podStartSLOduration=3.375501774 podStartE2EDuration="7.808535176s" totalImagesPullingTime="4.433033402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:03.694824294 +0000 UTC m=+7.786427294" watchObservedRunningTime="2026-11-02 02:47:03.808535176 +0000 UTC m=+7.900138174" Nov 02 02:47:04.516148 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:04.515735 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:04.516148 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:04.515879 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:04.516148 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:04.515966 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:04.516148 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:04.516047 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:04.599302 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:04.599258 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5sb5c" event={"ID":"91547ba7-4770-461b-a34b-0f3a46bdde43","Type":"ContainerStarted","Data":"49ffdb63613667d8db8571e7100b200ca2f1fba0c6cd224e5071484531818847"} Nov 02 02:47:05.011722 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:05.011631 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:05.012288 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:05.012110 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:05.127317 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:05.126442 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5sb5c" podStartSLOduration=3.634458626 podStartE2EDuration="9.126421766s" totalImagesPullingTime="5.49196314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:04.711226388 +0000 UTC m=+8.802829389" watchObservedRunningTime="2026-11-02 02:47:05.126421766 +0000 UTC m=+9.218024765" Nov 02 02:47:05.513555 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:05.513485 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:05.513730 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:05.513596 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:05.604558 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:05.604056 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" event={"ID":"60e09d2f-2971-476b-afd8-b3b37be1437b","Type":"ContainerStarted","Data":"a0a5062546a57c64f8e2eb1c3cd9955737ab442985c16c6cd9f5b4c6e84bb467"} Nov 02 02:47:06.074072 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:06.074041 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:06.074233 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:06.074087 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:06.074309 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.074257 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:06.074386 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.074315 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:14.074298381 +0000 UTC m=+18.165901373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:06.074459 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.074441 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:06.074508 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.074484 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret podName:f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c nodeName:}" failed. No retries permitted until 2026-11-02 02:47:14.074471177 +0000 UTC m=+18.166074167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret") pod "global-pull-secret-syncer-xhmws" (UID: "f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c") : object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:06.175025 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:06.174730 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:06.175025 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.174908 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 02:47:06.175025 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.174925 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 02:47:06.175025 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.174938 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k6dmj for pod openshift-network-diagnostics/network-check-target-d42js: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:06.175025 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.174992 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj podName:bb2c8c59-10f2-4f93-a4bd-1b37406fec26 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:14.1749735 +0000 UTC m=+18.266576491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6dmj" (UniqueName: "kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj") pod "network-check-target-d42js" (UID: "bb2c8c59-10f2-4f93-a4bd-1b37406fec26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:06.528487 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:06.527127 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:06.528487 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:06.527554 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:06.528487 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.528018 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:06.528487 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:06.528207 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:06.606253 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:06.606223 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 02:47:07.086494 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.086459 2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 02:47:07.508707 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.508556 2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T02:47:07.08647915Z","UUID":"8b7a032e-ef06-4482-afc5-d3751c419823","Handler":null,"Name":"","Endpoint":""} Nov 02 02:47:07.510301 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.510231 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 02:47:07.510301 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.510264 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 02:47:07.513316 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.513294 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:07.513452 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:07.513430 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:07.609734 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.609665 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfzxg" event={"ID":"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7","Type":"ContainerStarted","Data":"b9d5996853f249fcfd68ae3c721a704c3f80301597e1efeeb74a66fb9164cd3e"} Nov 02 02:47:07.612043 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.612015 2584 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="82b46a909150459e4c280bc3a6b395f365d01c5372f91166928764b8d6c108a7" exitCode=0 Nov 02 02:47:07.612194 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.612095 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerDied","Data":"82b46a909150459e4c280bc3a6b395f365d01c5372f91166928764b8d6c108a7"} Nov 02 02:47:07.614447 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.614047 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9kpxq" event={"ID":"42726530-4a21-425e-8598-05227546e93f","Type":"ContainerStarted","Data":"639deb2deaa870f120b56d81ac0b43609df62c404f2cd2861abfaaf8d5a70e82"} Nov 02 02:47:07.616418 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.616396 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" event={"ID":"60e09d2f-2971-476b-afd8-b3b37be1437b","Type":"ContainerStarted","Data":"238abce4b7a71c46b573218e3a8cbfefab166230c7271f15f47b7ca284a4033d"} Nov 02 02:47:07.724344 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.724297 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rfzxg" podStartSLOduration=3.5246370049999998 podStartE2EDuration="11.724282094s" totalImagesPullingTime="8.199645089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:07.724115154 +0000 UTC m=+11.815718152" watchObservedRunningTime="2026-11-02 02:47:07.724282094 +0000 UTC m=+11.815885092" Nov 02 02:47:07.952138 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:07.952090 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9kpxq" podStartSLOduration=3.838653422 podStartE2EDuration="11.952075861s" totalImagesPullingTime="8.113422439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:07.836467164 +0000 UTC m=+11.928070161" watchObservedRunningTime="2026-11-02 02:47:07.952075861 +0000 UTC m=+12.043678858" Nov 02 02:47:08.515511 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:08.515481 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:08.515651 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:08.515631 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:08.515711 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:08.515699 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:08.515812 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:08.515795 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:09.513695 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.513470 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:09.514100 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:09.513790 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:09.624015 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.623641 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zmp76" event={"ID":"41d1c58c-4c19-4212-934c-ecaf59792f37","Type":"ContainerStarted","Data":"d1adf2ad8c0f4e453df68e8051532eb4f5da9a728f5f1297be47af57269176d5"} Nov 02 02:47:09.625829 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.625802 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-76cvn" event={"ID":"9ba41133-2ab9-43ac-94d9-adfade07fee6","Type":"ContainerStarted","Data":"e438bc39a8b14f0eed482c1372d0400dfc6942af5b5ce92366d06a6c874bde24"} Nov 02 02:47:09.627960 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.627935 2584 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="149061d0556b5854570b5a70c0f3ff80b54310811532368d4d2de124b3864187" exitCode=0 Nov 02 02:47:09.628048 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.628014 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerDied","Data":"149061d0556b5854570b5a70c0f3ff80b54310811532368d4d2de124b3864187"} Nov 02 02:47:09.630428 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.630366 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" event={"ID":"60e09d2f-2971-476b-afd8-b3b37be1437b","Type":"ContainerStarted","Data":"df174b9028d05fddf954fb7c4aa1ac56ead85f79ac9861821fc6bc5ced61b4cd"} Nov 02 02:47:09.734557 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.734512 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zmp76" podStartSLOduration=5.30033107 podStartE2EDuration="13.734498967s" totalImagesPullingTime="8.434167897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:09.734161337 +0000 UTC m=+13.825764337" watchObservedRunningTime="2026-11-02 02:47:09.734498967 +0000 UTC m=+13.826101964" Nov 02 02:47:09.969389 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:09.969286 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" podStartSLOduration=4.399120371 podStartE2EDuration="13.969271807s" totalImagesPullingTime="9.570151436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:09.969065997 +0000 UTC m=+14.060668993" watchObservedRunningTime="2026-11-02 02:47:09.969271807 +0000 UTC m=+14.060874851" Nov 02 02:47:10.084482 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:10.084433 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-76cvn" podStartSLOduration=3.788436671 podStartE2EDuration="14.084414768s" totalImagesPullingTime="10.295978097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:10.084002104 +0000 UTC m=+14.175605101" watchObservedRunningTime="2026-11-02 02:47:10.084414768 +0000 UTC m=+14.176017766" Nov 02 02:47:10.513047 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:10.513017 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:10.513215 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:10.513056 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:10.513215 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:10.513137 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:10.513215 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:10.513206 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:11.512770 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:11.512730 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:11.513617 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:11.512845 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:11.637323 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:11.637277 2584 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="c9ee9b97411b834561567ab3ceece570cbb3b8dcd3eb8e2541565d9f6a05e063" exitCode=0 Nov 02 02:47:11.637505 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:11.637332 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerDied","Data":"c9ee9b97411b834561567ab3ceece570cbb3b8dcd3eb8e2541565d9f6a05e063"} Nov 02 02:47:12.513075 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:12.513038 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:12.513592 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:12.513170 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:12.513592 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:12.513263 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:12.513592 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:12.513389 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:13.513065 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:13.513032 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:13.513226 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:13.513194 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:14.135924 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.135645 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:14.136003 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.135939 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:14.136003 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.135778 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:14.136094 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.136042 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret podName:f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.136022325 +0000 UTC m=+34.227625305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret") pod "global-pull-secret-syncer-xhmws" (UID: "f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c") : object "kube-system"/"original-pull-secret" not registered Nov 02 02:47:14.136094 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.136046 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:14.136094 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.136080 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.136069987 +0000 UTC m=+34.227672966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 02:47:14.237071 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.237045 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:14.237203 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.237150 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 02:47:14.237203 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.237163 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 02:47:14.237203 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.237171 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k6dmj for pod openshift-network-diagnostics/network-check-target-d42js: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:14.237383 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.237212 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj podName:bb2c8c59-10f2-4f93-a4bd-1b37406fec26 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.237200558 +0000 UTC m=+34.328803533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6dmj" (UniqueName: "kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj") pod "network-check-target-d42js" (UID: "bb2c8c59-10f2-4f93-a4bd-1b37406fec26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 02:47:14.513139 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.513068 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:14.513286 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.513191 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:14.513286 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.513227 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:14.513682 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:14.513299 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:14.646159 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.646133 2584 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="a5b6211630fd4ec1be37004716e5a2ec53fc85e7c3a7cf9da262410a2badb29f" exitCode=0 Nov 02 02:47:14.646252 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.646198 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerDied","Data":"a5b6211630fd4ec1be37004716e5a2ec53fc85e7c3a7cf9da262410a2badb29f"} Nov 02 02:47:14.649179 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.649153 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"402d016676277a5e13242737c04c8c60cd2f49ccb89f29d5a8d908f9213aeccc"} Nov 02 02:47:14.649241 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.649226 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"de9730ba207fa95d25d228d1eae8be8e2d3f07b3df7329f249148d62e25009e9"} Nov 02 02:47:14.649288 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.649276 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"892b6d6c89ad09f741ca43c688cddf1bdb131092f09409e928a7172585b4207e"} Nov 02 02:47:14.649338 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.649325 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"a223ce1a065288b0b2dd9da94cfb461646ed7716325a3df4a18de6399a242367"} Nov 02 02:47:14.649409 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.649397 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"c29689dde16acec4281005a14a267125285537f2e84d281c914eb8599d201479"} Nov 02 02:47:14.649460 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:14.649445 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"bcbe4bd9fa37d88b146de96378a8dee1e7c629f196472d1990c97fc35ab5ae49"} Nov 02 02:47:15.513362 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:15.513319 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:15.513752 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:15.513428 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:15.654005 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:15.653972 2584 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="e53e3d9a48f3174ebd0c42ecf3c7d12725f5832a95f91345cbc54a68ec3a3063" exitCode=0 Nov 02 02:47:15.654144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:15.654017 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerDied","Data":"e53e3d9a48f3174ebd0c42ecf3c7d12725f5832a95f91345cbc54a68ec3a3063"} Nov 02 02:47:16.513757 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:16.513678 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:16.514205 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:16.513851 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:16.514205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:16.513890 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:16.514205 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:16.513969 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:16.659525 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:16.659497 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n56m9" event={"ID":"17e048e5-425c-43fe-8453-f4e30a51df0a","Type":"ContainerStarted","Data":"45828d3de0c7ba0a72ba41e4e3c889d957cb57d247a740aa2dfc96b399dfdf9c"} Nov 02 02:47:16.662647 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:16.662626 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"198d8e0fe836d4ec5220ee4dbb59041f29f42af6640e5da984647b2f382fa4f8"} Nov 02 02:47:16.775554 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:16.775482 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n56m9" podStartSLOduration=8.663948217 podStartE2EDuration="20.775466918s" totalImagesPullingTime="12.111518701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:16.774262398 +0000 UTC m=+20.865865394" watchObservedRunningTime="2026-11-02 02:47:16.775466918 +0000 UTC m=+20.867069914" Nov 02 02:47:17.513112 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:17.513079 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:17.513319 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:17.513200 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:18.513296 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.513264 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:18.514144 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:18.513410 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:18.514144 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.513465 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:18.514144 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:18.513576 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:18.670145 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.670073 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" event={"ID":"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5","Type":"ContainerStarted","Data":"91febe0cdac34a102ea90b4023862b0c45deca35a543fc3091dacd8198d63a95"} Nov 02 02:47:18.670407 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.670389 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:18.670484 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.670454 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:18.685199 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.685176 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:18.685301 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.685272 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:18.939578 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:18.939486 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" podStartSLOduration=7.896420979 podStartE2EDuration="22.93947408s" totalImagesPullingTime="15.043053101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:18.938920643 +0000 UTC m=+23.030523675" watchObservedRunningTime="2026-11-02 02:47:18.93947408 +0000 UTC m=+23.031077077" Nov 02 02:47:19.104169 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:19.104141 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:19.206983 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:19.206918 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:19.207118 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:19.207023 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 02:47:19.207472 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:19.207448 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:19.513109 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:19.513021 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:19.513261 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:19.513146 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:20.513550 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:20.513519 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:20.513987 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:20.513706 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:20.513987 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:20.513746 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:20.513987 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:20.513861 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:21.037233 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:21.037072 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhmws"] Nov 02 02:47:21.037396 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:21.037295 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:21.037459 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:21.037393 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:21.040649 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:21.040624 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cvzpz"] Nov 02 02:47:21.040791 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:21.040723 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:21.040852 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:21.040821 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:21.041278 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:21.041254 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d42js"] Nov 02 02:47:21.041397 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:21.041326 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:21.041449 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:21.041412 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:22.513171 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:22.513142 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:22.513171 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:22.513166 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:22.513595 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:22.513196 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:22.513595 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:22.513270 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:22.513595 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:22.513331 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:22.513595 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:22.513444 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:24.513486 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:24.513453 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:24.513486 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:24.513474 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:24.514134 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:24.513633 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:24.514134 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:24.513704 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:24.514134 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:24.513720 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:24.514134 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:24.513773 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:26.513342 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.513303 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:26.513818 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:26.513515 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d42js" podUID="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" Nov 02 02:47:26.515838 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.515801 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:26.515976 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:26.515954 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvzpz" podUID="b855c6cb-d565-4687-bd13-248d85111459" Nov 02 02:47:26.516040 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.516028 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:26.516143 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:26.516124 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhmws" podUID="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" Nov 02 02:47:26.708999 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.708945 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeReady" Nov 02 02:47:26.709085 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.709046 2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 02:47:26.821844 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.821055 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q69cv"] Nov 02 02:47:26.825448 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.825424 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-prxz2"] Nov 02 02:47:26.825558 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.825542 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:26.828131 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.828113 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldnnl"] Nov 02 02:47:26.828231 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.828215 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:26.830804 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.830790 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:26.870659 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.870640 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cb8549547-trw7f"] Nov 02 02:47:26.875035 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.875016 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-dn6x6"] Nov 02 02:47:26.875150 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.875133 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.878084 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.878068 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7"] Nov 02 02:47:26.878183 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.878130 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:26.881147 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.881132 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-djgk5"] Nov 02 02:47:26.882652 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.881479 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:26.885484 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.885372 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425"] Nov 02 02:47:26.885931 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.885912 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:26.889546 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.889527 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq"] Nov 02 02:47:26.889618 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.889604 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:26.892680 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.892662 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-648db889-4lftz"] Nov 02 02:47:26.892766 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.892729 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:26.895614 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.895591 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq"] Nov 02 02:47:26.895697 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.895657 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:26.898410 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.898338 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678859df4-mxqvr"] Nov 02 02:47:26.898499 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.898425 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:26.901204 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.901184 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2"] Nov 02 02:47:26.901373 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.901340 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:26.903538 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.903520 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-k6f8b"] Nov 02 02:47:26.903626 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.903611 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:26.906932 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.906788 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4"] Nov 02 02:47:26.906932 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.906919 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:26.909587 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.909570 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:26.919652 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.919633 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.919737 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.919638 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.921958 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.921938 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rgk6v"] Nov 02 02:47:26.926734 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.926716 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rvk96"] Nov 02 02:47:26.927146 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.927126 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:26.927723 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.927700 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 02:47:26.927831 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.927728 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 02:47:26.927906 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.927837 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hws9m\"" Nov 02 02:47:26.927970 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.927917 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 02:47:26.931093 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.931061 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q69cv"] Nov 02 02:47:26.931197 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.931102 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldnnl"] Nov 02 02:47:26.931197 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.931170 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:26.931964 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.931943 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-dn6x6"] Nov 02 02:47:26.935028 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935009 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.935028 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935042 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2klh\" (UniqueName: \"kubernetes.io/projected/76685eef-ba60-4361-9b35-c7291f159a8b-kube-api-access-c2klh\") pod \"kas-connection-checker-685895c7c9-prxz2\" (UID: \"76685eef-ba60-4361-9b35-c7291f159a8b\") " pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935070 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/210af9a6-abe4-4617-bf11-5ca72b7da8f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935108 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935136 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935177 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935212 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935236 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws78k\" (UniqueName: \"kubernetes.io/projected/e9bee798-853b-4ef3-b716-2e9bd203c71b-kube-api-access-ws78k\") pod \"kas-connection-checker-685895c7c9-q69cv\" (UID: \"e9bee798-853b-4ef3-b716-2e9bd203c71b\") " pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935264 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935286 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzlj\" (UniqueName: \"kubernetes.io/projected/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4-kube-api-access-8tzlj\") pod \"kas-connection-checker-685895c7c9-ldnnl\" (UID: \"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4\") " pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935341 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935385 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935425 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:26.936478 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.935877 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425"] Nov 02 02:47:26.937644 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.937622 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7"] Nov 02 02:47:26.937814 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.937668 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-djgk5"] Nov 02 02:47:26.939462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.939443 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cb8549547-trw7f"] Nov 02 02:47:26.978112 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978089 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 02:47:26.978307 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978286 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.978307 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978288 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.978307 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978291 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 02:47:26.978307 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978346 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978434 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978493 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978505 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978516 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978546 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978560 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978565 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978607 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978633 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978761 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978814 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.978912 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 02:47:26.979055 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.979049 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.984452 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.984432 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.984572 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.984468 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 02:47:26.989100 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.989080 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 02:47:26.989100 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.989104 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:26.989593 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:26.989182 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 02:47:27.027867 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.027827 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 02:47:27.027867 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.027860 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 02:47:27.028193 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.028101 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 02:47:27.034100 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.034080 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 02:47:27.036168 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036142 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.036291 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036180 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-tmp\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.036291 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036210 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.036414 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036325 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.036414 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036401 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.036512 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036429 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.036512 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036448 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4gq\" (UniqueName: \"kubernetes.io/projected/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-kube-api-access-js4gq\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.036609 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036506 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.036609 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036559 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.036682 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036617 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d03c81-d243-4970-9334-023b1af99f91-cert\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.036682 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036649 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.036682 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036675 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.036814 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036700 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.036814 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036786 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6t8\" (UniqueName: \"kubernetes.io/projected/872a3ac5-d06b-486c-a18f-99b640928ab3-kube-api-access-ck6t8\") pod \"network-check-source-6f78d59f99-z89m4\" (UID: \"872a3ac5-d06b-486c-a18f-99b640928ab3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:27.036876 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036825 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wk9\" (UniqueName: \"kubernetes.io/projected/971a0263-ee9a-4280-aa61-d385483ca8f0-kube-api-access-76wk9\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.036876 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036869 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.037022 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.036900 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.037022 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.036969 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:27.037022 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.036981 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:27.037174 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.037028 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.537011771 +0000 UTC m=+31.628614753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:27.037280 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037263 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.037336 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037287 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9cf6d2-1802-43d0-99ef-2480983f9f52-config-volume\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.037336 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037303 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-default-certificate\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.037336 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037330 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-trusted-ca\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037370 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgfj\" (UniqueName: \"kubernetes.io/projected/50d03c81-d243-4970-9334-023b1af99f91-kube-api-access-tfgfj\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037390 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037415 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/210af9a6-abe4-4617-bf11-5ca72b7da8f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037436 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037454 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037469 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77r8\" (UniqueName: \"kubernetes.io/projected/4ee35072-3ed9-4c17-9d67-714aec085b58-kube-api-access-d77r8\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037506 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gghqz\" (UniqueName: \"kubernetes.io/projected/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-kube-api-access-gghqz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037526 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-config\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037540 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-stats-auth\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.037574 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037566 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6hz\" (UniqueName: \"kubernetes.io/projected/d25eaef0-b0ac-4326-925b-aaef13840661-kube-api-access-vt6hz\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037595 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037623 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037664 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws78k\" (UniqueName: \"kubernetes.io/projected/e9bee798-853b-4ef3-b716-2e9bd203c71b-kube-api-access-ws78k\") pod \"kas-connection-checker-685895c7c9-q69cv\" (UID: \"e9bee798-853b-4ef3-b716-2e9bd203c71b\") " pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037696 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037715 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037746 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzlj\" (UniqueName: \"kubernetes.io/projected/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4-kube-api-access-8tzlj\") pod \"kas-connection-checker-685895c7c9-ldnnl\" (UID: \"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4\") " pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037769 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/e3744822-b0ed-4aeb-9ab9-f5401a3905f0-kube-api-access-d9xfh\") pod \"volume-data-source-validator-58c9f5f46d-7l425\" (UID: \"e3744822-b0ed-4aeb-9ab9-f5401a3905f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037786 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037801 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-service-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037841 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037864 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9cf6d2-1802-43d0-99ef-2480983f9f52-tmp-dir\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037888 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037910 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037927 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlzv\" (UniqueName: \"kubernetes.io/projected/4c9cf6d2-1802-43d0-99ef-2480983f9f52-kube-api-access-zqlzv\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.038068 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037955 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2klh\" (UniqueName: \"kubernetes.io/projected/76685eef-ba60-4361-9b35-c7291f159a8b-kube-api-access-c2klh\") pod \"kas-connection-checker-685895c7c9-prxz2\" (UID: \"76685eef-ba60-4361-9b35-c7291f159a8b\") " pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037971 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c94082a0-1953-42aa-9615-7820c50317eb-config\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.037992 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971a0263-ee9a-4280-aa61-d385483ca8f0-serving-cert\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038501 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038561 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c94082a0-1953-42aa-9615-7820c50317eb-serving-cert\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038592 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrxk\" (UniqueName: \"kubernetes.io/projected/c94082a0-1953-42aa-9615-7820c50317eb-kube-api-access-9hrxk\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038697 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038724 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17b96028-853b-4750-a2b3-09658c06aa2e-serving-cert\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.038869 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038780 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:27.039190 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038893 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-snapshots\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.039190 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.038940 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnfb\" (UniqueName: \"kubernetes.io/projected/17b96028-853b-4750-a2b3-09658c06aa2e-kube-api-access-nhnfb\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.039317 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.039299 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.039410 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.039389 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.039462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.039416 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/210af9a6-abe4-4617-bf11-5ca72b7da8f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:27.042031 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.042011 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.042342 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.042324 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.077639 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.077608 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 02:47:27.077773 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.077686 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 02:47:27.078004 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.077984 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 02:47:27.078096 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078068 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 02:47:27.078096 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078082 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8tvz9\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078105 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gvcps\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078098 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078130 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078139 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078166 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078076 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 02:47:27.078205 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078167 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078216 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c7bng\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078230 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6b8nz\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078260 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078299 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078328 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078371 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078392 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wsjnf\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078394 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6ddg2\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078404 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-msd8h\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078431 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-82426\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078438 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gdqg4\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078442 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2g54f\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078454 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078460 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgnd5\"" Nov 02 02:47:27.078543 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078469 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vbwmz\"" Nov 02 02:47:27.079076 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078752 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 02:47:27.079076 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078784 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2"] Nov 02 02:47:27.079076 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.078788 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 02:47:27.079076 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.078892 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:27.079076 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.078945 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.578928234 +0000 UTC m=+31.670531226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:27.079468 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.079455 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-prxz2"] Nov 02 02:47:27.080118 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.080102 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rgk6v"] Nov 02 02:47:27.081258 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.081245 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq"] Nov 02 02:47:27.081889 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.081875 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvk96"] Nov 02 02:47:27.082484 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.082472 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678859df4-mxqvr"] Nov 02 02:47:27.083043 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.083031 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-k6f8b"] Nov 02 02:47:27.085451 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.085432 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq"] Nov 02 02:47:27.088128 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.088109 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-648db889-4lftz"] Nov 02 02:47:27.088511 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.088491 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.088885 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.088872 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4"] Nov 02 02:47:27.097454 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.097435 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzlj\" (UniqueName: \"kubernetes.io/projected/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4-kube-api-access-8tzlj\") pod \"kas-connection-checker-685895c7c9-ldnnl\" (UID: \"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4\") " pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:27.097588 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.097571 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2klh\" (UniqueName: \"kubernetes.io/projected/76685eef-ba60-4361-9b35-c7291f159a8b-kube-api-access-c2klh\") pod \"kas-connection-checker-685895c7c9-prxz2\" (UID: \"76685eef-ba60-4361-9b35-c7291f159a8b\") " pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:27.097934 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.097918 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.098048 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.098033 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws78k\" (UniqueName: \"kubernetes.io/projected/e9bee798-853b-4ef3-b716-2e9bd203c71b-kube-api-access-ws78k\") pod \"kas-connection-checker-685895c7c9-q69cv\" (UID: \"e9bee798-853b-4ef3-b716-2e9bd203c71b\") " pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:27.098110 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.098086 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.137741 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.137721 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:27.140026 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140009 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrxk\" (UniqueName: \"kubernetes.io/projected/c94082a0-1953-42aa-9615-7820c50317eb-kube-api-access-9hrxk\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.140093 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.140093 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140050 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17b96028-853b-4750-a2b3-09658c06aa2e-serving-cert\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.140093 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140077 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-snapshots\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.140190 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnfb\" (UniqueName: \"kubernetes.io/projected/17b96028-853b-4750-a2b3-09658c06aa2e-kube-api-access-nhnfb\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.140271 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140242 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.140400 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140284 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-tmp\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.140400 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.140400 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140380 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140447 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.140465 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140476 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4gq\" (UniqueName: \"kubernetes.io/projected/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-kube-api-access-js4gq\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140504 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.140534 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.640516543 +0000 UTC m=+31.732119518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.140548 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:27.140561 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140555 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140583 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d03c81-d243-4970-9334-023b1af99f91-cert\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140606 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.140627 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.640609593 +0000 UTC m=+31.732212585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140662 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140689 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140715 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-snapshots\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140730 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6t8\" (UniqueName: \"kubernetes.io/projected/872a3ac5-d06b-486c-a18f-99b640928ab3-kube-api-access-ck6t8\") pod \"network-check-source-6f78d59f99-z89m4\" (UID: \"872a3ac5-d06b-486c-a18f-99b640928ab3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140759 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wk9\" (UniqueName: \"kubernetes.io/projected/971a0263-ee9a-4280-aa61-d385483ca8f0-kube-api-access-76wk9\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140808 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140836 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9cf6d2-1802-43d0-99ef-2480983f9f52-config-volume\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.140878 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140861 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-default-certificate\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140890 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-trusted-ca\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140915 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgfj\" (UniqueName: \"kubernetes.io/projected/50d03c81-d243-4970-9334-023b1af99f91-kube-api-access-tfgfj\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140942 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140970 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77r8\" (UniqueName: \"kubernetes.io/projected/4ee35072-3ed9-4c17-9d67-714aec085b58-kube-api-access-d77r8\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140979 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.140997 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghqz\" (UniqueName: \"kubernetes.io/projected/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-kube-api-access-gghqz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-config\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141059 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-stats-auth\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141088 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6hz\" (UniqueName: \"kubernetes.io/projected/d25eaef0-b0ac-4326-925b-aaef13840661-kube-api-access-vt6hz\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141119 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.141143 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.64112363 +0000 UTC m=+31.732726618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141168 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.141192 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141195 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.142610 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.141237 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.641223479 +0000 UTC m=+31.732826468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141257 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/e3744822-b0ed-4aeb-9ab9-f5401a3905f0-kube-api-access-d9xfh\") pod \"volume-data-source-validator-58c9f5f46d-7l425\" (UID: \"e3744822-b0ed-4aeb-9ab9-f5401a3905f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141289 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-service-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141380 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-tmp\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141407 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141488 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9cf6d2-1802-43d0-99ef-2480983f9f52-tmp-dir\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141522 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141584 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlzv\" (UniqueName: \"kubernetes.io/projected/4c9cf6d2-1802-43d0-99ef-2480983f9f52-kube-api-access-zqlzv\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141858 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-service-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141620 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c94082a0-1953-42aa-9615-7820c50317eb-config\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141935 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971a0263-ee9a-4280-aa61-d385483ca8f0-serving-cert\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141145 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.141970 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.142005 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c94082a0-1953-42aa-9615-7820c50317eb-serving-cert\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.143245 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.142240 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9cf6d2-1802-43d0-99ef-2480983f9f52-config-volume\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.142462 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.142735 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-trusted-ca\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.142761 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.142777 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.142804 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17b96028-853b-4750-a2b3-09658c06aa2e-serving-cert\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.142818 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.642803022 +0000 UTC m=+31.734406013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.143077 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9cf6d2-1802-43d0-99ef-2480983f9f52-tmp-dir\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.143155 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.143217 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:27.643203039 +0000 UTC m=+31.734806014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.143443 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.144027 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.143845 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-config\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.144626 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.144183 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c94082a0-1953-42aa-9615-7820c50317eb-config\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.144626 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.144244 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.144626 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.144319 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:27.145772 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.144999 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.145772 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.145276 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.145772 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.145469 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-default-certificate\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.145772 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.145565 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c94082a0-1953-42aa-9615-7820c50317eb-serving-cert\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.146271 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.146228 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.149089 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.146862 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-stats-auth\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.149089 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.147309 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971a0263-ee9a-4280-aa61-d385483ca8f0-serving-cert\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.149089 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.147333 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d03c81-d243-4970-9334-023b1af99f91-cert\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.149089 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.148416 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:27.217131 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.217058 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnfb\" (UniqueName: \"kubernetes.io/projected/17b96028-853b-4750-a2b3-09658c06aa2e-kube-api-access-nhnfb\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.217648 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.217627 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.218216 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.218192 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4gq\" (UniqueName: \"kubernetes.io/projected/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-kube-api-access-js4gq\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.218431 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.218413 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrxk\" (UniqueName: \"kubernetes.io/projected/c94082a0-1953-42aa-9615-7820c50317eb-kube-api-access-9hrxk\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.223531 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.223510 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.224053 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.224016 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77r8\" (UniqueName: \"kubernetes.io/projected/4ee35072-3ed9-4c17-9d67-714aec085b58-kube-api-access-d77r8\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.225725 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.225704 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghqz\" (UniqueName: \"kubernetes.io/projected/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-kube-api-access-gghqz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.225877 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.225858 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/e3744822-b0ed-4aeb-9ab9-f5401a3905f0-kube-api-access-d9xfh\") pod \"volume-data-source-validator-58c9f5f46d-7l425\" (UID: \"e3744822-b0ed-4aeb-9ab9-f5401a3905f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:27.227029 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.227009 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6t8\" (UniqueName: \"kubernetes.io/projected/872a3ac5-d06b-486c-a18f-99b640928ab3-kube-api-access-ck6t8\") pod \"network-check-source-6f78d59f99-z89m4\" (UID: \"872a3ac5-d06b-486c-a18f-99b640928ab3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:27.227673 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.227654 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wk9\" (UniqueName: \"kubernetes.io/projected/971a0263-ee9a-4280-aa61-d385483ca8f0-kube-api-access-76wk9\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.227778 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.227758 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6hz\" (UniqueName: \"kubernetes.io/projected/d25eaef0-b0ac-4326-925b-aaef13840661-kube-api-access-vt6hz\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.229272 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.229231 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgfj\" (UniqueName: \"kubernetes.io/projected/50d03c81-d243-4970-9334-023b1af99f91-kube-api-access-tfgfj\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.232678 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.232478 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:27.233096 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.232878 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlzv\" (UniqueName: \"kubernetes.io/projected/4c9cf6d2-1802-43d0-99ef-2480983f9f52-kube-api-access-zqlzv\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.254827 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.254474 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:27.258948 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.258848 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:27.269670 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.267906 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:27.273621 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.273444 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:27.506373 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.506286 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:27.512575 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.512546 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:27.558818 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.558790 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:27.559196 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.558954 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:27.559196 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.558975 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:27.559196 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.559041 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.559022075 +0000 UTC m=+32.650625067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:27.564299 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.564275 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-prxz2"] Nov 02 02:47:27.565146 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.565115 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q69cv"] Nov 02 02:47:27.570886 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.570856 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76685eef_ba60_4361_9b35_c7291f159a8b.slice/crio-96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd WatchSource:0}: Error finding container 96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd: Status 404 returned error can't find the container with id 96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd Nov 02 02:47:27.585821 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.585781 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldnnl"] Nov 02 02:47:27.597770 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.597740 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c5e81a_fe80_4a82_bfbc_c094e1f8fde4.slice/crio-58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac WatchSource:0}: Error finding container 58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac: Status 404 returned error can't find the container with id 58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac Nov 02 02:47:27.660411 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.660287 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:27.660411 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.660335 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:27.660411 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.660378 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.660789 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.660767327 +0000 UTC m=+32.752370317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.660892 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.660929 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.660917712 +0000 UTC m=+32.752520701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.660988 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.661021 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.66101126 +0000 UTC m=+32.752614237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.661075 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.661104 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.661094271 +0000 UTC m=+32.752697261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.661124 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.661159 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.661270 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:27.661445 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.661313 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:27.664612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664414 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:27.664612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664466 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.664451966 +0000 UTC m=+32.756054960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:27.664612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664529 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:27.664612 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664579 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.664554466 +0000 UTC m=+32.756157455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:27.665027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664935 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:27.665027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664954 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:27.665027 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:27.664997 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:28.664983638 +0000 UTC m=+32.756586627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:27.671950 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.671857 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-k6f8b"] Nov 02 02:47:27.677469 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.677448 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq"] Nov 02 02:47:27.678105 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.678077 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971a0263_ee9a_4280_aa61_d385483ca8f0.slice/crio-7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d WatchSource:0}: Error finding container 7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d: Status 404 returned error can't find the container with id 7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d Nov 02 02:47:27.679907 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.679870 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2"] Nov 02 02:47:27.683113 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.683085 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94082a0_1953_42aa_9615_7820c50317eb.slice/crio-a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209 WatchSource:0}: Error finding container a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209: Status 404 returned error can't find the container with id a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209 Nov 02 02:47:27.691688 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.691645 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-prxz2" event={"ID":"76685eef-ba60-4361-9b35-c7291f159a8b","Type":"ContainerStarted","Data":"96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd"} Nov 02 02:47:27.693178 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.693152 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q69cv" event={"ID":"e9bee798-853b-4ef3-b716-2e9bd203c71b","Type":"ContainerStarted","Data":"fcf7b919c1218c3a62e1c860c00a39313d49d33846208428d16cc4e5149f2e97"} Nov 02 02:47:27.694536 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.694513 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" event={"ID":"971a0263-ee9a-4280-aa61-d385483ca8f0","Type":"ContainerStarted","Data":"7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d"} Nov 02 02:47:27.696381 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.696342 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6abdd8_3b6c_40a6_8cd5_20229ff88bb0.slice/crio-1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57 WatchSource:0}: Error finding container 1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57: Status 404 returned error can't find the container with id 1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57 Nov 02 02:47:27.696959 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.696934 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" event={"ID":"c94082a0-1953-42aa-9615-7820c50317eb","Type":"ContainerStarted","Data":"a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209"} Nov 02 02:47:27.701301 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.701275 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" event={"ID":"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4","Type":"ContainerStarted","Data":"58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac"} Nov 02 02:47:27.707475 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.707454 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4"] Nov 02 02:47:27.708127 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.708103 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872a3ac5_d06b_486c_a18f_99b640928ab3.slice/crio-d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9 WatchSource:0}: Error finding container d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9: Status 404 returned error can't find the container with id d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9 Nov 02 02:47:27.708387 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.708370 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rgk6v"] Nov 02 02:47:27.896887 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.896859 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-djgk5"] Nov 02 02:47:27.899392 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.899366 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b96028_853b_4750_a2b3_09658c06aa2e.slice/crio-84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5 WatchSource:0}: Error finding container 84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5: Status 404 returned error can't find the container with id 84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5 Nov 02 02:47:27.941375 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:27.941332 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425"] Nov 02 02:47:27.944576 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:27.944555 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3744822_b0ed_4aeb_9ab9_f5401a3905f0.slice/crio-38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89 WatchSource:0}: Error finding container 38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89: Status 404 returned error can't find the container with id 38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89 Nov 02 02:47:28.513716 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.513677 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:28.513716 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.513705 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:28.513716 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.513714 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:28.565490 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.565462 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 02:47:28.565765 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.565708 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6md5s\"" Nov 02 02:47:28.565765 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.565720 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gkth\"" Nov 02 02:47:28.565765 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.565713 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 02:47:28.565765 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.565760 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x92pp\"" Nov 02 02:47:28.575156 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.575138 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:28.575266 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.575254 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:28.575312 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.575267 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:28.575312 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.575306 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.57529485 +0000 UTC m=+34.666897825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:28.676121 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676096 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:28.676228 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676167 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:28.676228 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676193 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:28.676228 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676210 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:28.676228 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676223 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676241 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676272 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676254556 +0000 UTC m=+34.767857552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676296 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676333 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676399 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676407 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676391327 +0000 UTC m=+34.767994306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676339 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676337 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:28.676433 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676436 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676421298 +0000 UTC m=+34.768024295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:28.676718 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676408 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:28.676718 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676464 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676451292 +0000 UTC m=+34.768054269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:28.676718 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676482 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676474045 +0000 UTC m=+34.768077029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:28.676718 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676530 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676522562 +0000 UTC m=+34.768125539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:28.676718 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.676631 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:28.676880 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676733 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:28.676880 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:28.676792 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:30.676781169 +0000 UTC m=+34.768384145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:28.704409 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.704388 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" event={"ID":"e3744822-b0ed-4aeb-9ab9-f5401a3905f0","Type":"ContainerStarted","Data":"38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89"} Nov 02 02:47:28.706036 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.706011 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q69cv" event={"ID":"e9bee798-853b-4ef3-b716-2e9bd203c71b","Type":"ContainerStarted","Data":"4f6b521c03a788edc83620dae79de3487204af05e77ddd8fb95ace912b74cbce"} Nov 02 02:47:28.707104 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.707085 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-djgk5" event={"ID":"17b96028-853b-4750-a2b3-09658c06aa2e","Type":"ContainerStarted","Data":"84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5"} Nov 02 02:47:28.708150 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.708114 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" event={"ID":"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0","Type":"ContainerStarted","Data":"1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57"} Nov 02 02:47:28.709444 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.709421 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" event={"ID":"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4","Type":"ContainerStarted","Data":"0a0712d818a2f9dad66bb5f8d7406a2d54fff3314be19fd4ccf4983ff1304926"} Nov 02 02:47:28.710500 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.710481 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rgk6v" event={"ID":"50d03c81-d243-4970-9334-023b1af99f91","Type":"ContainerStarted","Data":"454ac2a3ad86f3274775f191950d3114885c41463ff78c0a85de30e49f6d0e1d"} Nov 02 02:47:28.711462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.711441 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" event={"ID":"872a3ac5-d06b-486c-a18f-99b640928ab3","Type":"ContainerStarted","Data":"d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9"} Nov 02 02:47:28.712604 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:28.712585 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-prxz2" event={"ID":"76685eef-ba60-4361-9b35-c7291f159a8b","Type":"ContainerStarted","Data":"d78b8f409c2f16359968f5841ec62bc06dfee1ad2ce8e78b03b3cb3588f379df"} Nov 02 02:47:29.038720 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:29.038640 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" podStartSLOduration=439.038622672 podStartE2EDuration="7m19.038622672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:29.038382764 +0000 UTC m=+33.129985764" watchObservedRunningTime="2026-11-02 02:47:29.038622672 +0000 UTC m=+33.130225666" Nov 02 02:47:29.149626 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:29.149238 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-q69cv" podStartSLOduration=439.149222101 podStartE2EDuration="7m19.149222101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:29.149000535 +0000 UTC m=+33.240603536" watchObservedRunningTime="2026-11-02 02:47:29.149222101 +0000 UTC m=+33.240825094" Nov 02 02:47:29.262473 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:29.262418 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-prxz2" podStartSLOduration=439.262400807 podStartE2EDuration="7m19.262400807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:29.260532224 +0000 UTC m=+33.352135227" watchObservedRunningTime="2026-11-02 02:47:29.262400807 +0000 UTC m=+33.354003806" Nov 02 02:47:30.191382 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.191236 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:30.192009 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.191394 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:30.192009 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.191599 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 02:47:30.192009 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.191664 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:48:02.191645074 +0000 UTC m=+66.283248051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : secret "metrics-daemon-secret" not found Nov 02 02:47:30.204786 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.204761 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:30.293207 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.292546 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:30.300701 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.300575 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:30.354201 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.353992 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:30.370110 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.370032 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:30.597639 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.596897 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:30.597639 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.597161 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:30.597639 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.597177 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:30.597639 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.597235 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.597217068 +0000 UTC m=+38.688820062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:30.698553 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698514 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:30.698723 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698615 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:30.698723 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698691 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:30.698723 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698722 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:30.698901 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698738 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:30.698901 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698761 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:30.698901 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.698782 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:30.699061 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.698913 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:30.699061 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.698975 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.698957963 +0000 UTC m=+38.790560961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:30.699373 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699343 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:30.699453 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699408 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.699393755 +0000 UTC m=+38.790996744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:30.699514 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699458 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:30.699514 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699466 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:30.699514 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699494 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.699482587 +0000 UTC m=+38.791085567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:30.699664 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699556 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:30.699664 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699589 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.699579293 +0000 UTC m=+38.791182269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:30.699664 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699636 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:30.699664 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699655 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.699649339 +0000 UTC m=+38.791252314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:30.699860 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699698 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:30.699860 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699726 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.699716205 +0000 UTC m=+38.791319193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:30.699860 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:30.699779 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:34.699773737 +0000 UTC m=+38.791376712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:30.811174 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.811104 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d42js"] Nov 02 02:47:30.860072 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:30.860039 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhmws"] Nov 02 02:47:30.923752 ip-10-0-137-74 kubenswrapper[2584]: W1102 02:47:30.923610 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f5b5ce_fe6f_479e_ab0a_fc40bfaa1b5c.slice/crio-8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf WatchSource:0}: Error finding container 8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf: Status 404 returned error can't find the container with id 8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf Nov 02 02:47:31.726731 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:31.726696 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" event={"ID":"e3744822-b0ed-4aeb-9ab9-f5401a3905f0","Type":"ContainerStarted","Data":"22960194d81899a320899172e2f215685ecaaf7ab0d11e6e82f941e583527b55"} Nov 02 02:47:31.732200 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:31.732113 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d42js" event={"ID":"bb2c8c59-10f2-4f93-a4bd-1b37406fec26","Type":"ContainerStarted","Data":"5980f4c704ea616cf3ce505bf5a63811dcedb155d77a1800ccc2a105ca19b1f7"} Nov 02 02:47:31.735681 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:31.735609 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhmws" event={"ID":"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c","Type":"ContainerStarted","Data":"8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf"} Nov 02 02:47:31.839036 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:31.838980 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" podStartSLOduration=414.835162452 podStartE2EDuration="6m57.838967084s" totalImagesPullingTime="3.003804632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:31.838764427 +0000 UTC m=+35.930367422" watchObservedRunningTime="2026-11-02 02:47:31.838967084 +0000 UTC m=+35.930570082" Nov 02 02:47:33.743681 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:33.743654 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/0.log" Nov 02 02:47:33.744022 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:33.743694 2584 generic.go:356] "Generic (PLEG): container finished" podID="971a0263-ee9a-4280-aa61-d385483ca8f0" containerID="0289eb0a8e6ea548bbe4b4e82cf3c9f0c6e6c1ed578412ad6c22c94338d92638" exitCode=255 Nov 02 02:47:33.744022 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:33.743745 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" event={"ID":"971a0263-ee9a-4280-aa61-d385483ca8f0","Type":"ContainerDied","Data":"0289eb0a8e6ea548bbe4b4e82cf3c9f0c6e6c1ed578412ad6c22c94338d92638"} Nov 02 02:47:33.744204 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:33.744056 2584 scope.go:122] "RemoveContainer" containerID="0289eb0a8e6ea548bbe4b4e82cf3c9f0c6e6c1ed578412ad6c22c94338d92638" Nov 02 02:47:34.649259 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.649220 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:34.649532 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.649368 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:34.649532 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.649385 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:34.649532 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.649441 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.649427276 +0000 UTC m=+46.741030267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:34.749121 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.749081 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-djgk5" event={"ID":"17b96028-853b-4750-a2b3-09658c06aa2e","Type":"ContainerStarted","Data":"b2f5b5002254c35420fea1621da20eb1b8220e69200838f096395d8e187dd0ca"} Nov 02 02:47:34.749660 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.749636 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:34.749810 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.749687 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:34.749810 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.749714 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:34.749810 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.749738 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:34.749956 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.749824 2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:34.749956 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.749861 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.749843479 +0000 UTC m=+46.841446469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:34.749956 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.749865 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:34.749956 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.749938 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.749915588 +0000 UTC m=+46.841518577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:34.749956 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.749956 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.749946602 +0000 UTC m=+46.841549581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.749980 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.750043 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750065 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750103 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.750089156 +0000 UTC m=+46.841692138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.750067 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750111 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750119 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750147 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.750138045 +0000 UTC m=+46.841741025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750174 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:34.750202 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750201 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.750191896 +0000 UTC m=+46.841794918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:34.750668 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750247 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:34.750668 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.750300 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.750286596 +0000 UTC m=+46.841889577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:34.752000 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.751967 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" event={"ID":"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0","Type":"ContainerStarted","Data":"c9a24da030f3ea94237038f114aec7de0f48939399f6a4a2b0bc425a16f2ec44"} Nov 02 02:47:34.753823 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.753805 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/1.log" Nov 02 02:47:34.754261 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.754243 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/0.log" Nov 02 02:47:34.754347 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.754283 2584 generic.go:356] "Generic (PLEG): container finished" podID="971a0263-ee9a-4280-aa61-d385483ca8f0" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" exitCode=255 Nov 02 02:47:34.754414 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.754378 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" event={"ID":"971a0263-ee9a-4280-aa61-d385483ca8f0","Type":"ContainerDied","Data":"a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e"} Nov 02 02:47:34.754450 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.754437 2584 scope.go:122] "RemoveContainer" containerID="0289eb0a8e6ea548bbe4b4e82cf3c9f0c6e6c1ed578412ad6c22c94338d92638" Nov 02 02:47:34.754599 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.754584 2584 scope.go:122] "RemoveContainer" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" Nov 02 02:47:34.754766 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:34.754744 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-k6f8b_openshift-console-operator(971a0263-ee9a-4280-aa61-d385483ca8f0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" Nov 02 02:47:34.755850 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.755825 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" event={"ID":"c94082a0-1953-42aa-9615-7820c50317eb","Type":"ContainerStarted","Data":"d9d39a83d52fb021c0fd068ae34724fe666d36bd64e7fe75c065e16df0666e9d"} Nov 02 02:47:34.757614 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.757592 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rgk6v" event={"ID":"50d03c81-d243-4970-9334-023b1af99f91","Type":"ContainerStarted","Data":"3dbaf27f267d8254ccff85d218186f4f5c5356d45937fab909a701121d89eadc"} Nov 02 02:47:34.863607 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.863555 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-djgk5" podStartSLOduration=415.094646342 podStartE2EDuration="7m0.863541937s" totalImagesPullingTime="5.768895595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:34.862273386 +0000 UTC m=+38.953876387" watchObservedRunningTime="2026-11-02 02:47:34.863541937 +0000 UTC m=+38.955144955" Nov 02 02:47:34.975653 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:34.975602 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" podStartSLOduration=415.022404049 podStartE2EDuration="7m0.975586061s" totalImagesPullingTime="5.953182012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:34.974004749 +0000 UTC m=+39.065607758" watchObservedRunningTime="2026-11-02 02:47:34.975586061 +0000 UTC m=+39.067189077" Nov 02 02:47:35.087575 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.087530 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rgk6v" podStartSLOduration=2.755959759 podStartE2EDuration="9.087512907s" totalImagesPullingTime="6.331553148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:35.085769081 +0000 UTC m=+39.177372079" watchObservedRunningTime="2026-11-02 02:47:35.087512907 +0000 UTC m=+39.179115903" Nov 02 02:47:35.258759 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.258676 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5jsrl"] Nov 02 02:47:35.277416 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.277388 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.320857 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.320797 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" podStartSLOduration=415.35774209 podStartE2EDuration="7m1.320778699s" totalImagesPullingTime="5.963036609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:35.320039777 +0000 UTC m=+39.411642775" watchObservedRunningTime="2026-11-02 02:47:35.320778699 +0000 UTC m=+39.412381697" Nov 02 02:47:35.322951 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.322930 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jsrl"] Nov 02 02:47:35.330102 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.330037 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 02:47:35.330512 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.330484 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 02:47:35.330857 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.330803 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvbct\"" Nov 02 02:47:35.356284 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.356257 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60a499f-81af-466f-9be9-a581feee8523-crio-socket\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.356427 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.356346 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.356427 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.356399 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.356555 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.356537 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnr8z\" (UniqueName: \"kubernetes.io/projected/d60a499f-81af-466f-9be9-a581feee8523-kube-api-access-xnr8z\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.356770 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.356750 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60a499f-81af-466f-9be9-a581feee8523-data-volume\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.457708 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.457662 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60a499f-81af-466f-9be9-a581feee8523-data-volume\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.457938 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.457767 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60a499f-81af-466f-9be9-a581feee8523-crio-socket\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.457938 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.457806 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.457938 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.457832 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.457938 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.457933 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnr8z\" (UniqueName: \"kubernetes.io/projected/d60a499f-81af-466f-9be9-a581feee8523-kube-api-access-xnr8z\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.458134 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.458007 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60a499f-81af-466f-9be9-a581feee8523-data-volume\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.458134 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.458067 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 02:47:35.458134 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.458114 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:35.958098925 +0000 UTC m=+40.049701900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : configmap "kube-rbac-proxy" not found Nov 02 02:47:35.458294 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.458156 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60a499f-81af-466f-9be9-a581feee8523-crio-socket\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.458294 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.458212 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:35.458294 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.458247 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:35.95823982 +0000 UTC m=+40.049842794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:35.517090 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.517025 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnr8z\" (UniqueName: \"kubernetes.io/projected/d60a499f-81af-466f-9be9-a581feee8523-kube-api-access-xnr8z\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.763131 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.763027 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" event={"ID":"872a3ac5-d06b-486c-a18f-99b640928ab3","Type":"ContainerStarted","Data":"fed894cbfd6b1510b06089d2c6f8b6c1457aa3f0d67be89c423ed313b445b06c"} Nov 02 02:47:35.764918 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.764883 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d42js" event={"ID":"bb2c8c59-10f2-4f93-a4bd-1b37406fec26","Type":"ContainerStarted","Data":"b3d0bcb4818778f1b980010dd331721b24c8c9c774d0fdb80d4d14a641476fd3"} Nov 02 02:47:35.765071 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.764944 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:35.766669 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.766646 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/1.log" Nov 02 02:47:35.766967 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.766947 2584 scope.go:122] "RemoveContainer" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" Nov 02 02:47:35.767163 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.767115 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-k6f8b_openshift-console-operator(971a0263-ee9a-4280-aa61-d385483ca8f0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" Nov 02 02:47:35.876189 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.876137 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" podStartSLOduration=409.765181467 podStartE2EDuration="6m56.876123771s" totalImagesPullingTime="7.110942304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:35.874705328 +0000 UTC m=+39.966308326" watchObservedRunningTime="2026-11-02 02:47:35.876123771 +0000 UTC m=+39.967726768" Nov 02 02:47:35.964656 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.964624 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.964656 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:35.964661 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:35.964900 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.964752 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 02:47:35.964900 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.964822 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:36.96480314 +0000 UTC m=+41.056406129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : configmap "kube-rbac-proxy" not found Nov 02 02:47:35.964900 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.964839 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:35.964900 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:35.964892 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:36.96487495 +0000 UTC m=+41.056477932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:36.102708 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:36.102657 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d42js" podStartSLOduration=36.051849971 podStartE2EDuration="40.102642315s" totalImagesPullingTime="4.050792344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:35.988395171 +0000 UTC m=+40.079998169" watchObservedRunningTime="2026-11-02 02:47:36.102642315 +0000 UTC m=+40.194245329" Nov 02 02:47:36.975983 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:36.975945 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:36.976462 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:36.975992 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:36.976462 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:36.976075 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 02:47:36.976462 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:36.976141 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:36.976462 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:36.976163 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:38.976141163 +0000 UTC m=+43.067744142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : configmap "kube-rbac-proxy" not found Nov 02 02:47:36.976462 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:36.976188 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:38.976177659 +0000 UTC m=+43.067780634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:37.150198 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.150149 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8xt2_3ecd4ba6-46cc-4f76-8984-492c35ad4f40/konnectivity-agent/0.log" limit=1024 Nov 02 02:47:37.259058 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.258993 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:37.259058 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.259044 2584 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:37.259341 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.259329 2584 scope.go:122] "RemoveContainer" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" Nov 02 02:47:37.259533 ip-10-0-137-74 kubenswrapper[2584]: E1102 02:47:37.259515 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-k6f8b_openshift-console-operator(971a0263-ee9a-4280-aa61-d385483ca8f0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" Nov 02 02:47:37.429207 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.429181 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v8xt2_3ecd4ba6-46cc-4f76-8984-492c35ad4f40/konnectivity-agent/0.log" limit=1024 Nov 02 02:47:37.775866 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.775833 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhmws" event={"ID":"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c","Type":"ContainerStarted","Data":"97da89633dc27eae531ac097ab649ce2b21a0e81f8b2f542ca9b18a1227439cd"} Nov 02 02:47:37.889933 ip-10-0-137-74 kubenswrapper[2584]: I1102 02:47:37.889828 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xhmws" podStartSLOduration=35.216726087 podStartE2EDuration="41.889814313s" totalImagesPullingTime="6.673088226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:47:37.888922698 +0000 UTC m=+41.980525692" watchObservedRunningTime="2026-11-02 02:47:37.889814313 +0000 UTC m=+41.981417309" Nov 02 02:47:37.917299 ip-10-0-137-74 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 02:47:37.932726 ip-10-0-137-74 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 02:47:37.932880 ip-10-0-137-74 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 02:47:37.932927 ip-10-0-137-74 systemd[1]: kubelet.service: Consumed 2.852s CPU time. Nov 02 02:47:37.943882 ip-10-0-137-74 systemd[1]: Starting Kubernetes Kubelet... Nov 02 02:47:38.034003 ip-10-0-137-74 kubenswrapper[4868]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 02:47:38.034003 ip-10-0-137-74 kubenswrapper[4868]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 02:47:38.034003 ip-10-0-137-74 kubenswrapper[4868]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 02:47:38.034003 ip-10-0-137-74 kubenswrapper[4868]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 02:47:38.036011 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.035940 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.036011 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036011 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036015 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036020 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036023 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036026 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036028 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036031 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036042 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036045 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036047 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036050 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036053 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036056 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036058 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036061 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036063 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036066 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036069 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036071 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036074 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.036082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036077 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036080 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036083 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036086 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036089 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036092 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036094 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036097 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036100 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036103 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036105 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036108 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036110 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036113 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036115 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036118 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036120 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036123 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036126 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036129 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.036597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036131 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036134 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036136 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036139 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036141 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036144 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036146 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036149 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036152 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036154 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036157 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036160 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036162 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036165 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036168 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036171 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036174 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036176 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036180 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.037082 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036184 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036188 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036202 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036206 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036209 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036212 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036215 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036217 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036220 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036222 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036225 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036228 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036237 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036240 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036243 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036245 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036248 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036250 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036252 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.037566 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036255 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036258 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036261 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036263 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036266 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036269 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036272 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036275 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036277 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036280 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036282 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036285 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036287 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036290 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036292 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036295 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036298 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036300 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036303 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036305 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.038035 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036308 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036310 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036313 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036316 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036319 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036321 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.036330 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037104 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037111 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037115 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037119 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037122 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037125 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037128 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037131 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037134 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037136 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037140 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037142 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037145 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.038534 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037148 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037150 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037152 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037155 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037158 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037161 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037163 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037166 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037168 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037171 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037173 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037176 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037178 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037181 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037184 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037186 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037189 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037193 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037197 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.039022 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037201 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037204 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037207 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037210 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037212 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037215 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037217 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037220 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037222 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037224 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037227 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037230 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037232 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037235 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037237 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037240 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037242 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037245 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037247 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037249 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.039550 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037252 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037254 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037259 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037262 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037265 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037267 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037270 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037272 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037276 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037278 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037281 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037283 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037286 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037289 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037292 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037294 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037297 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037299 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037302 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037304 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.040049 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037307 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037309 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037312 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037314 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037316 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037319 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037321 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037324 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037326 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037329 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037332 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037334 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037337 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037340 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037344 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037347 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037363 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037366 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037369 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.040552 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037372 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037374 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037378 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037382 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037385 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037388 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037390 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037393 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037396 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037399 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037401 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037404 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037406 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037409 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037413 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037901 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037907 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037911 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037913 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.041017 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037916 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037919 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037923 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037927 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037931 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037934 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037937 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037940 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037945 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037948 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037950 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037953 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037956 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037958 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037961 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037963 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037966 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037969 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037971 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.041488 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037974 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037977 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037980 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037983 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037985 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037988 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037991 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037993 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037996 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.037999 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038001 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038003 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038006 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038009 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038011 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038014 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038017 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038019 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038023 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.041965 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038025 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038028 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038032 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038035 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038037 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038040 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038043 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038046 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038050 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038053 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038057 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038061 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038064 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038066 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038069 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038073 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038076 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038078 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038081 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.042675 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038084 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038086 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038090 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038093 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038096 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038099 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038101 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038104 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038106 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038109 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038112 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038114 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038117 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038119 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038122 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038125 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038128 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038130 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038133 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038135 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.043306 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038138 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038140 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038143 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038145 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038148 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038150 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038154 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038156 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038159 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038162 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038165 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038167 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038170 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038172 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038176 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038179 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038181 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038184 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038186 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038189 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.043814 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038191 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038194 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038196 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038199 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038202 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038287 4868 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038299 4868 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038305 4868 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038310 4868 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038315 4868 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038318 4868 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038322 4868 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038326 4868 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038329 4868 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038332 4868 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038336 4868 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038339 4868 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038342 4868 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038345 4868 flags.go:64] FLAG: --cgroup-root="" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038361 4868 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038364 4868 flags.go:64] FLAG: --client-ca-file="" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038367 4868 flags.go:64] FLAG: --cloud-provider="external" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038370 4868 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038374 4868 flags.go:64] FLAG: --cluster-domain="" Nov 02 02:47:38.044332 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038377 4868 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038380 4868 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038387 4868 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038390 4868 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038394 4868 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038398 4868 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038401 4868 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038404 4868 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038407 4868 flags.go:64] FLAG: --contention-profiling="false" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038410 4868 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038413 4868 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038416 4868 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038419 4868 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038423 4868 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038426 4868 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038429 4868 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038432 4868 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038436 4868 flags.go:64] FLAG: --enable-server="true" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038438 4868 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038443 4868 flags.go:64] FLAG: --event-burst="100" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038446 4868 flags.go:64] FLAG: --event-qps="50" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038449 4868 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038452 4868 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038455 4868 flags.go:64] FLAG: --eviction-hard="" Nov 02 02:47:38.044942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038459 4868 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038462 4868 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038465 4868 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038468 4868 flags.go:64] FLAG: --eviction-soft="" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038471 4868 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038474 4868 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038477 4868 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038487 4868 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038490 4868 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038493 4868 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038498 4868 flags.go:64] FLAG: --feature-gates="" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038506 4868 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038510 4868 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038513 4868 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038516 4868 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038519 4868 flags.go:64] FLAG: --healthz-port="10248" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038522 4868 flags.go:64] FLAG: --help="false" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038525 4868 flags.go:64] FLAG: --hostname-override="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038529 4868 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038532 4868 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038535 4868 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038538 4868 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038542 4868 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 02:47:38.045567 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038546 4868 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038549 4868 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038552 4868 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038555 4868 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038558 4868 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038561 4868 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038564 4868 flags.go:64] FLAG: --kube-reserved="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038567 4868 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038570 4868 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038573 4868 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038576 4868 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038578 4868 flags.go:64] FLAG: --lock-file="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038581 4868 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038584 4868 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038587 4868 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038594 4868 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038597 4868 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038601 4868 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038603 4868 flags.go:64] FLAG: --logging-format="text" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038608 4868 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038611 4868 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038614 4868 flags.go:64] FLAG: --manifest-url="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038617 4868 flags.go:64] FLAG: --manifest-url-header="" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038621 4868 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038624 4868 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 02:47:38.046148 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038628 4868 flags.go:64] FLAG: --max-pods="110" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038631 4868 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038634 4868 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038637 4868 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038640 4868 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038643 4868 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038646 4868 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038649 4868 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038656 4868 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038659 4868 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038662 4868 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038665 4868 flags.go:64] FLAG: --pod-cidr="" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038668 4868 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038671 4868 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038674 4868 flags.go:64] FLAG: --pods-per-core="0" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038676 4868 flags.go:64] FLAG: --port="10250" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038680 4868 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038683 4868 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ca8720f3b8cbc55d" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038686 4868 flags.go:64] FLAG: --qos-reserved="" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038689 4868 flags.go:64] FLAG: --read-only-port="10255" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038692 4868 flags.go:64] FLAG: --register-node="true" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038695 4868 flags.go:64] FLAG: --register-with-taints="" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038700 4868 flags.go:64] FLAG: --registry-burst="10" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038703 4868 flags.go:64] FLAG: --registry-qps="5" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038705 4868 flags.go:64] FLAG: --reserved-cpus="" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038709 4868 flags.go:64] FLAG: --reserved-memory="" Nov 02 02:47:38.046795 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038714 4868 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038717 4868 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038720 4868 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038723 4868 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038726 4868 flags.go:64] FLAG: --runonce="false" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038729 4868 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038732 4868 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038735 4868 flags.go:64] FLAG: --seccomp-default="false" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038738 4868 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038741 4868 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038744 4868 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038747 4868 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038750 4868 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038753 4868 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038756 4868 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038758 4868 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038761 4868 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038765 4868 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038767 4868 flags.go:64] FLAG: --system-cgroups="" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038770 4868 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038776 4868 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038779 4868 flags.go:64] FLAG: --tls-cert-file="" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038782 4868 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038785 4868 flags.go:64] FLAG: --tls-min-version="" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038788 4868 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 02:47:38.047441 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038791 4868 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038794 4868 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038797 4868 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038801 4868 flags.go:64] FLAG: --v="2" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038806 4868 flags.go:64] FLAG: --version="false" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038810 4868 flags.go:64] FLAG: --vmodule="" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038814 4868 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.038819 4868 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038941 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038945 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038947 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038950 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038953 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038955 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038958 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038960 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038963 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038965 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038968 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038970 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038973 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.048040 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038976 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038978 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038981 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038983 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038986 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038989 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038991 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038994 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038996 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.038999 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039001 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039004 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039006 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039009 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039013 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039015 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039018 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039021 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039025 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.048643 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039028 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039031 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039034 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039036 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039039 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039042 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039044 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039047 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039049 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039052 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039054 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039057 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039059 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039062 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039064 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039067 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039070 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039072 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039076 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039080 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.049193 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039082 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039085 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039088 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039090 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039092 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039095 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039097 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039101 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039104 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039106 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039109 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039112 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039115 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039118 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039121 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039123 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039126 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039129 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039131 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039134 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.049721 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039137 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039139 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039142 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039145 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039147 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039150 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039152 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039155 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039157 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039160 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039162 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039165 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039168 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039170 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039172 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039175 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039178 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039180 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039183 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039187 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.050263 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039189 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039192 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039194 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039198 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039202 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039205 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039210 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039213 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039215 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039218 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039220 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039223 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039226 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039228 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.039253 4868 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039287 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.050947 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039291 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039294 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039297 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039300 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039303 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039307 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039310 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039313 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039316 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039318 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039321 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039323 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039326 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039329 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039331 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039335 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039338 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039340 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039343 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.051390 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039346 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039361 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039364 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039367 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039370 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039373 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039376 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039378 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039381 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039384 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039386 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039389 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039392 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039394 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039397 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039399 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039402 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039404 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039407 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039409 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.051895 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039412 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039414 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039417 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039420 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039422 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039425 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039427 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039430 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039434 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039436 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039439 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039443 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039447 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039451 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039454 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039457 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039460 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039463 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039466 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.052408 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039468 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039471 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039473 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039476 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039478 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039481 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039484 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039486 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039489 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039491 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039494 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039496 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039499 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039501 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039504 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039506 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039509 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039512 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039514 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.052910 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039517 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039520 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039524 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039526 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039529 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039531 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039534 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039552 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039556 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039559 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039562 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039565 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039568 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039571 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039574 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039578 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039582 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039586 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039590 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039594 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.053421 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039598 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039602 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039606 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039610 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039614 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039617 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039621 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.039623 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.044270 4868 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.044285 4868 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.044303 4868 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.044308 4868 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044381 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044388 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044393 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044396 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044398 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044401 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044404 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044407 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.053904 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044410 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044412 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044415 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044417 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044420 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044422 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044425 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044429 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044431 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044434 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044436 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044439 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044441 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044444 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044447 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044449 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044452 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044454 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044457 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.054406 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044460 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044462 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044465 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044467 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044470 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044472 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044475 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044478 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044481 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044483 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044486 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044489 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044492 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044494 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044497 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044500 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044502 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044505 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044507 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044510 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.054855 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044513 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044515 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044518 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044521 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044523 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044526 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044528 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044531 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044533 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044536 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044538 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044541 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044543 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044546 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044548 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044551 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044553 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044556 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044559 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044561 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.055515 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044564 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044567 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044569 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044572 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044575 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044577 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044580 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044582 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044585 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044588 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044590 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044593 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044596 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044599 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044601 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044604 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044606 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044610 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044614 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.056264 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044617 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044620 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044623 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044625 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044628 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044630 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044633 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044635 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044638 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044640 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044643 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044645 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044648 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044651 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044653 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044656 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044659 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044661 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044664 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.056926 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044666 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.044694 4868 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044833 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044838 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044841 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044844 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044847 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044850 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044852 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044855 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044857 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044860 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044863 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044865 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044868 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044871 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.057400 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044873 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044876 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044879 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044882 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044884 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044887 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044889 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044892 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044896 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044900 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044903 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044906 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044909 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044912 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044915 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044917 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044920 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044923 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044928 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.057828 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044930 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044933 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044935 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044938 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044940 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044943 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044946 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044949 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044952 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044954 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044957 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044960 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044962 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044965 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044967 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044970 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044972 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044975 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044977 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044980 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.058277 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044982 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044985 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044987 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044990 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044992 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044995 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.044997 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045000 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045003 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045005 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045008 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045012 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045015 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045017 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045020 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045023 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045026 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045028 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045031 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.058871 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045034 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045036 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045039 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045041 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045043 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045046 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045049 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045051 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045054 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045057 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045059 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045062 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045064 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045067 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045069 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045072 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045076 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045079 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045082 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.059312 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045084 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045087 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045089 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045092 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045094 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045099 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045101 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045104 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045107 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045110 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045112 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045117 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045120 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045124 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045126 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.045150 4868 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 02:47:38.059787 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045182 4868 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045186 4868 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045189 4868 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045193 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045196 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045199 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045201 4868 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045205 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045207 4868 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045210 4868 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045213 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045215 4868 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045218 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045221 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045223 4868 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045226 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045229 4868 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045232 4868 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045234 4868 feature_gate.go:386] unrecognized feature gate: Example Nov 02 02:47:38.060305 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045237 4868 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045239 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045244 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045246 4868 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045249 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045251 4868 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045254 4868 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045257 4868 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045259 4868 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045264 4868 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045266 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045269 4868 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045272 4868 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045275 4868 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045278 4868 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045280 4868 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045283 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045285 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045289 4868 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 02:47:38.060873 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045292 4868 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045295 4868 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045298 4868 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045300 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045303 4868 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045306 4868 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045309 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045311 4868 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045313 4868 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045316 4868 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045319 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045321 4868 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045324 4868 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045326 4868 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045329 4868 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045333 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045335 4868 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045338 4868 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045341 4868 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 02:47:38.061508 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045343 4868 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045345 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045362 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045367 4868 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045370 4868 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045373 4868 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045376 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045380 4868 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045383 4868 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045386 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045388 4868 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045391 4868 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045393 4868 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045396 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045399 4868 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045401 4868 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045404 4868 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045407 4868 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045409 4868 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 02:47:38.061993 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045412 4868 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045415 4868 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045419 4868 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045422 4868 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045425 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045428 4868 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045430 4868 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045432 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045435 4868 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045439 4868 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045442 4868 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045444 4868 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045447 4868 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045449 4868 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045452 4868 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045455 4868 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045459 4868 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045461 4868 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045464 4868 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045467 4868 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 02:47:38.062543 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045471 4868 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045473 4868 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045476 4868 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045478 4868 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045481 4868 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045484 4868 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045486 4868 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045489 4868 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045491 4868 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:38.045494 4868 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.045540 4868 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.046514 4868 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.046564 4868 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.046734 4868 server.go:1014] "Starting client certificate rotation" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.046875 4868 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.046921 4868 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 02:41:56 +0000 UTC" deadline="2028-05-29 20:11:06.834333283 +0000 UTC" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.046955 4868 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13793h23m28.78738009s" Nov 02 02:47:38.063043 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.047165 4868 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 02:47:38.063537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.048183 4868 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 02:47:38.063537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.049386 4868 log.go:25] "Validated CRI v1 runtime API" Nov 02 02:47:38.063537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.050920 4868 log.go:25] "Validated CRI v1 image API" Nov 02 02:47:38.063537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.052013 4868 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 02:47:38.063537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.053795 4868 fs.go:135] Filesystem UUIDs: map[00a52bc1-5abc-4a8b-a2e6-06c32f0beb90:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe09c98c-e710-4552-8994-e21049e9cc1c:/dev/nvme0n1p3] Nov 02 02:47:38.063740 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.053809 4868 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/454ac2a3ad86f3274775f191950d3114885c41463ff78c0a85de30e49f6d0e1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/454ac2a3ad86f3274775f191950d3114885c41463ff78c0a85de30e49f6d0e1d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5980f4c704ea616cf3ce505bf5a63811dcedb155d77a1800ccc2a105ca19b1f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5980f4c704ea616cf3ce505bf5a63811dcedb155d77a1800ccc2a105ca19b1f7/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcf7b919c1218c3a62e1c860c00a39313d49d33846208428d16cc4e5149f2e97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcf7b919c1218c3a62e1c860c00a39313d49d33846208428d16cc4e5149f2e97/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17b96028-853b-4750-a2b3-09658c06aa2e/volumes/kubernetes.io~projected/kube-api-access-nhnfb:{mountpoint:/var/lib/kubelet/pods/17b96028-853b-4750-a2b3-09658c06aa2e/volumes/kubernetes.io~projected/kube-api-access-nhnfb major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17b96028-853b-4750-a2b3-09658c06aa2e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/17b96028-853b-4750-a2b3-09658c06aa2e/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17e048e5-425c-43fe-8453-f4e30a51df0a/volumes/kubernetes.io~projected/kube-api-access-9kgmn:{mountpoint:/var/lib/kubelet/pods/17e048e5-425c-43fe-8453-f4e30a51df0a/volumes/kubernetes.io~projected/kube-api-access-9kgmn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/201c03d0-ed5b-4ccb-b4c2-534fe47511c4/volumes/kubernetes.io~projected/kube-api-access-js4gq:{mountpoint:/var/lib/kubelet/pods/201c03d0-ed5b-4ccb-b4c2-534fe47511c4/volumes/kubernetes.io~projected/kube-api-access-js4gq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volumes/kubernetes.io~projected/kube-api-access-d67nx:{mountpoint:/var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volumes/kubernetes.io~projected/kube-api-access-d67nx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ecd4ba6-46cc-4f76-8984-492c35ad4f40/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3ecd4ba6-46cc-4f76-8984-492c35ad4f40/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41d1c58c-4c19-4212-934c-ecaf59792f37/volumes/kubernetes.io~projected/kube-api-access-6hnsj:{mountpoint:/var/lib/kubelet/pods/41d1c58c-4c19-4212-934c-ecaf59792f37/volumes/kubernetes.io~projected/kube-api-access-6hnsj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42726530-4a21-425e-8598-05227546e93f/volumes/kubernetes.io~projected/kube-api-access-bf47p:{mountpoint:/var/lib/kubelet/pods/42726530-4a21-425e-8598-05227546e93f/volumes/kubernetes.io~projected/kube-api-access-bf47p major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c9cf6d2-1802-43d0-99ef-2480983f9f52/volumes/kubernetes.io~projected/kube-api-access-zqlzv:{mountpoint:/var/lib/kubelet/pods/4c9cf6d2-1802-43d0-99ef-2480983f9f52/volumes/kubernetes.io~projected/kube-api-access-zqlzv major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ee35072-3ed9-4c17-9d67-714aec085b58/volumes/kubernetes.io~projected/kube-api-access-d77r8:{mountpoint:/var/lib/kubelet/pods/4ee35072-3ed9-4c17-9d67-714aec085b58/volumes/kubernetes.io~projected/kube-api-access-d77r8 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d03c81-d243-4970-9334-023b1af99f91/volumes/kubernetes.io~projected/kube-api-access-tfgfj:{mountpoint:/var/lib/kubelet/pods/50d03c81-d243-4970-9334-023b1af99f91/volumes/kubernetes.io~projected/kube-api-access-tfgfj major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d03c81-d243-4970-9334-023b1af99f91/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/50d03c81-d243-4970-9334-023b1af99f91/volumes/kubernetes.io~secret/cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~projected/kube-api-access-8452z:{mountpoint:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~projected/kube-api-access-8452z major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e09d2f-2971-476b-afd8-b3b37be1437b/volumes/kubernetes.io~projected/kube-api-access-6jb6m:{mountpoint:/var/lib/kubelet/pods/60e09d2f-2971-476b-afd8-b3b37be1437b/volumes/kubernetes.io~projected/kube-api-access-6jb6m major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76685eef-ba60-4361-9b35-c7291f159a8b/volumes/kubernetes.io~projected/kube-api-access-c2klh:{mountpoint:/var/lib/kubelet/pods/76685eef-ba60-4361-9b35-c7291f159a8b/volumes/kubernetes.io~projected/kube-api-access-c2klh major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/872a3ac5-d06b-486c-a18f-99b640928ab3/volumes/kubernetes.io~projected/kube-api-access-ck6t8:{mountpoint:/var/lib/kubelet/pods/872a3ac5-d06b-486c-a18f-99b640928ab3/volumes/kubernetes.io~projected/kube-api-access-ck6t8 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91547ba7-4770-461b-a34b-0f3a46bdde43/volumes/kubernetes.io~projected/kube-api-access-xvhhr:{mountpoint:/var/lib/kubelet/pods/91547ba7-4770-461b-a34b-0f3a46bdde43/volumes/kubernetes.io~projected/kube-api-access-xvhhr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971a0263-ee9a-4280-aa61-d385483ca8f0/volumes/kubernetes.io~projected/kube-api-access-76wk9:{mountpoint:/var/lib/kubelet/pods/971a0263-ee9a-4280-aa61-d385483ca8f0/volumes/kubernetes.io~projected/kube-api-access-76wk9 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971a0263-ee9a-4280-aa61-d385483ca8f0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/971a0263-ee9a-4280-aa61-d385483ca8f0/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~projected/kube-api-access-5bd78:{mountpoint:/var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~projected/kube-api-access-5bd78 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~projected/kube-api-access-mmthm:{mountpoint:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~projected/kube-api-access-mmthm major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b855c6cb-d565-4687-bd13-248d85111459/volumes/kubernetes.io~projected/kube-api-access-grznx:{mountpoint:/var/lib/kubelet/pods/b855c6cb-d565-4687-bd13-248d85111459/volumes/kubernetes.io~projected/kube-api-access-grznx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/volumes/kubernetes.io~projected/kube-api-access-gghqz:{mountpoint:/var/lib/kubelet/pods/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/volumes/kubernetes.io~projected/kube-api-access-gghqz major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb2c8c59-10f2-4f93-a4bd-1b37406fec26/volumes/kubernetes.io~projected/kube-api-access-k6dmj:{mountpoint:/var/lib/kubelet/pods/bb2c8c59-10f2-4f93-a4bd-1b37406fec26/volumes/kubernetes.io~projected/kube-api-access-k6dmj major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c94082a0-1953-42aa-9615-7820c50317eb/volumes/kubernetes.io~projected/kube-api-access-9hrxk:{mountpoint:/var/lib/kubelet/pods/c94082a0-1953-42aa-9615-7820c50317eb/volumes/kubernetes.io~projected/kube-api-access-9hrxk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c94082a0-1953-42aa-9615-7820c50317eb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c94082a0-1953-42aa-9615-7820c50317eb/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~projected/kube-api-access-vt6hz:{mountpoint:/var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~projected/kube-api-access-vt6hz major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4/volumes/kubernetes.io~projected/kube-api-access-8tzlj:{mountpoint:/var/lib/kubelet/pods/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4/volumes/kubernetes.io~projected/kube-api-access-8tzlj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d60a499f-81af-466f-9be9-a581feee8523/volumes/kubernetes.io~projected/kube-api-access-xnr8z:{mountpoint:/var/lib/kubelet/pods/d60a499f-81af-466f-9be9-a581feee8523/volumes/kubernetes.io~projected/kube-api-access-xnr8z major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3744822-b0ed-4aeb-9ab9-f5401a3905f0/volumes/kubernetes.io~projected/kube-api-access-d9xfh:{mountpoint:/var/lib/kubelet/pods/e3744822-b0ed-4aeb-9ab9-f5401a3905f0/volumes/kubernetes.io~projected/kube-api-access-d9xfh major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9bee798-853b-4ef3-b716-2e9bd203c71b/volumes/kubernetes.io~projected/kube-api-access-ws78k:{mountpoint:/var/lib/kubelet/pods/e9bee798-853b-4ef3-b716-2e9bd203c71b/volumes/kubernetes.io~projected/kube-api-access-ws78k major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7/volumes/kubernetes.io~projected/kube-api-access-xz78w:{mountpoint:/var/lib/kubelet/pods/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7/volumes/kubernetes.io~projected/kube-api-access-xz78w major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/401b8e630a8c363a191ccd05514f9121dcd189ba6dabe9dd66f41e1c5f2bfbf7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/acaee5a5da92e6995cdc0f8f19c570173908cd62ed3185a371bebed3ea814251/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/77c4593a62ab6f19bf23566541a72a6fa66df7a1135636227dfa2f9543dc9c3f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/71af4cab50d702cf3583ea76ee8774b11bb96afa4a1eb0d3f225ec15320d6aa7/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bc51c0fe2c0ecd5a7aa432a90f51ccd3d7437cc9d508c5ccd3024d5eb8653e2d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/dfef889305324abed91aea7e2569831cac3f04008eb18950ee33d070ba9366dd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bd068f113597e7fbc84eeabf0addfdb621a86c360cdb63761407272a270fc646/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1ef73061629eb8d81fd7c5b3351990aeb4e113d4738b710e84567c6512cbef35/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/abf4c9819878f6dc2c0d026a9ddfc16fdd0dbdb635891dd0b4c95d44fac3e7e8/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/01bc846e9b4abe62d6abcc64a798ee8a41e6b252da7704eebed3a2e94fd73c7c/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/e26ce9ba1ce0ef5477a5e6efcf4929c1f3ed760a84ff58c3241c61b73455524f/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/37439d816dc3d2bfb012b5583bde3fd39aafc7472f0e1ee448bf70c343c8779c/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ccfdb6f2f0593a78a7c996d4f3e4fb212f8fbd1f6a3fa5822611e68839e2401a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f0ac906078086b997e90801731d7f6bc2dca66180eb5eea5436e1ab76dd8a55c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/215c428c97c96b604630c8e75396409bb372dd36d86f531599a93686f5d774ed/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/6ca7f1b25591772791a65cbea9cc3c5ac802c8354dcbf477c78a8da91b704e79/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c2e1c7278537d5e9a314d2445970530767a403ab6b9c0480bf311c405bba3c41/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5672a0e72a95cf79ca436833dc7924a5e0cfd4100a8b1421c88cddb8c47bd319/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6a03c28936d967f216ad17c92e5e3b1e215ba69edbea5304c43edd66d50fe3bd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2f8eaabeeef3f2977cdaf740c271cf6bd7661c00e02c6bdf50e2677b02767482/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b59f8d3d96fcb369f29a6d604e445e9138e3fa35ec9c9fe6eddb537166ba3e41/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c4ed31a250a22079e992ea6c18e7b43f7cb28de6f6980d3cad5450a74bb9eba1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8b99b31cf1fd955012ef74b92b44dcf8a25bd36d9144e5f18e20fe4bfbfc9745/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/dbe9bbd4079546a06307a23bab0975a9ff63153c5e1618426fd67de135581bc8/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/70d9586cbee425c84212ac14437390b33bfef49526915d7f3e75de0544852f8b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/994c3107e51058a812f29e1057c9ef244da9a38d02ad5147e56888e56da49174/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c72950f5048c4bb26a5ac09d79f5a80e01b4df130da5f2c629ba00bfc731dd2a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/4814ec88abce2d519fd31a62b5c5ce19b88a150e4ff9b34f26f9093490a78d3f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/733d6ffd6956d5c7d9fb1267d39af292b16f6d5cd18884b43badabdc47e088e9/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/fee5d9a7c0387d97273a5550b44c7041b8192d1c93ade43447a6170994f65924/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/f91ac8bfecf3009347e4902fd500f200a788f3f5023539907768ed6011ba0223/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/ee6e3aadd5d644c604934d92d512179b454b04130d6ad9de6d2b73363fdd4f4e/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/8d9c83f3e413c88e3c0a0075ed07985e71526abf9b6eeae81e22a3543c162a22/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/25462626e79df00eb671d7f46385b4fef6138c993a4666a7e13dc133bfd37be6/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/2c61d331327c3af58b1d9dd82f50728ac8766c3e9d1dc67d3d2ca4daa1187193/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d03e49d6d04a28d69e43c5054da535718f506846104d46f5931d38aaea9edb7d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/73722cbdff92e9aa87c61c0f3793de448353f0fcbde662d7fabb1c2fae4c70b0/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/14903768e6e080d6e9ff4323b4a2b4d58f0666b9dbd1cf84c3d966e99cf45dc1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/c4d9f3af89148823b0c78cc9e9f9d34acf9f8ff9b8ad1587052bd0069995ba59/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e9a7d97c4eaf23dd2837be74d713cd62d5687f64f0a7543e4101409e4fc69d08/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/6cb4b441cc17498b8d8885a2ef33120c43792d2e101c3e4a9e616cd4cafc1a6a/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/e2f374ef5399528fa1dea4a7471e759e021446057ca8a9250f6cb232d5b16c7c/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/6f3d5b6fc86331faf00abd14ddd61782ede2d8499252728c7dc6b5b9f2ee3dcf/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/a7a3224e0ecd9d3673852d62091ab2c5bbbf3643a659ddd81a4d888dde91a17d/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/29056a17f412715a870e8f9f71fdc3d9da1c484ec25fffa07ba2c7b2165a0134/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3536b1843f2a0a3287b44b4a72408a2d6567d5b07c562549ac95231a3be5fd02/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/044cc22af8fbf27f14d57c271d54040c63634bcb12b6967b55b26c027339ee64/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f72cc1f796f46ff44a805b5a8082005e3b06062fd833ba4a50fc0e9705da55ed/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/9c26ea6089aa0a61d46973834db7854c4bf5e3020f4878f01a57e21d68829a85/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ac5da03e23c40ae7baf4ca514bd95cd29cf4bd11ea479eca2ad3c0cb057294a3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0b80ff23c4f98372a758321ebbf336e21034452ef462f20d1d6bca379e78c36a/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d9cd7cc786c6959a9838f0f7215f66386be3b0484910aa90e625a29f09d1647f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c57334215b149ccf1dfdb3f451e2efa9268ec50077fc8097ea433988a5cd0ccd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/17bbf7bfd2229383b6295f4855fd82aeb5e95f6159beb2ac9e4fd491561bcd0e/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.061958 4868 manager.go:217] Machine: {Timestamp:2026-11-02 02:47:38.061274185 +0000 UTC m=+0.056975146 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099698 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d28ca222015037014e343675fb299 SystemUUID:ec2d28ca-2220-1503-7014-e343675fb299 BootID:d41b5709-d316-4fe1-b0b1-462a60c78450 Filesystems:[{Device:/var/lib/kubelet/pods/41d1c58c-4c19-4212-934c-ecaf59792f37/volumes/kubernetes.io~projected/kube-api-access-6hnsj DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91547ba7-4770-461b-a34b-0f3a46bdde43/volumes/kubernetes.io~projected/kube-api-access-xvhhr DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17b96028-853b-4750-a2b3-09658c06aa2e/volumes/kubernetes.io~projected/kube-api-access-nhnfb DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1018915fa064c96065e99e6bda875480850932cc711184a9d95879f7a8de6a57/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/454ac2a3ad86f3274775f191950d3114885c41463ff78c0a85de30e49f6d0e1d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/201c03d0-ed5b-4ccb-b4c2-534fe47511c4/volumes/kubernetes.io~projected/kube-api-access-js4gq DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c9cf6d2-1802-43d0-99ef-2480983f9f52/volumes/kubernetes.io~projected/kube-api-access-zqlzv DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b855c6cb-d565-4687-bd13-248d85111459/volumes/kubernetes.io~projected/kube-api-access-grznx DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~projected/kube-api-access-5bd78 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d6f05caa89f570963349789353abe276ac8545fc9f9c35bb5ef31bce59907e4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3407a8fc1fdd02411a1d68bdd26ba1f3cd9658db8ae410afe1a202d623c0834/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d4ec28883bab07b395cb6f2eacab531de0d7b9bb03f712f35bc4e195f325852/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d60a499f-81af-466f-9be9-a581feee8523/volumes/kubernetes.io~projected/kube-api-access-xnr8z DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~projected/kube-api-access-mmthm DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/971a0263-ee9a-4280-aa61-d385483ca8f0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcf7b919c1218c3a62e1c860c00a39313d49d33846208428d16cc4e5149f2e97/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96baecf9a45445c2741e477a22dd84a4b67b800f41a52ebca3a52e1f7ab3a4dd/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0c521964940da809dc0fa829c1ebee179f41971241d9f36ae4b780b118118c8/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76685eef-ba60-4361-9b35-c7291f159a8b/volumes/kubernetes.io~projected/kube-api-access-c2klh DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42726530-4a21-425e-8598-05227546e93f/volumes/kubernetes.io~projected/kube-api-access-bf47p DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/872a3ac5-d06b-486c-a18f-99b640928ab3/volumes/kubernetes.io~projected/kube-api-access-ck6t8 DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17e048e5-425c-43fe-8453-f4e30a51df0a/volumes/kubernetes.io~projected/kube-api-access-9kgmn DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~projected/kube-api-access-vt6hz DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb2c8c59-10f2-4f93-a4bd-1b37406fec26/volumes/kubernetes.io~projected/kube-api-access-k6dmj DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50d03c81-d243-4970-9334-023b1af99f91/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4/volumes/kubernetes.io~projected/kube-api-access-8tzlj DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38470b6e56f69577d88e8f61f2422078fe70477c4f91cf1c123dbd862f1c8d89/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50d03c81-d243-4970-9334-023b1af99f91/volumes/kubernetes.io~projected/kube-api-access-tfgfj DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17b96028-853b-4750-a2b3-09658c06aa2e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58b38b8f37fbebc4ab206a2b42b576f8b9b30a6e04cd2dc566acb18ec01278ac/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c94082a0-1953-42aa-9615-7820c50317eb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3744822-b0ed-4aeb-9ab9-f5401a3905f0/volumes/kubernetes.io~projected/kube-api-access-d9xfh DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e09d2f-2971-476b-afd8-b3b37be1437b/volumes/kubernetes.io~projected/kube-api-access-6jb6m DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84f8d19e6f1a828f5a7e3b119bae407d4ee645c1253359be9323df604beaa4c5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~projected/kube-api-access-8452z DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7461b2d6073283c25dd4b4929050cc520fb133d9cb570b52d93218e0dd5ded9d/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ba41133-2ab9-43ac-94d9-adfade07fee6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ee35072-3ed9-4c17-9d67-714aec085b58/volumes/kubernetes.io~projected/kube-api-access-d77r8 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volumes/kubernetes.io~projected/kube-api-access-d67nx DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9bee798-853b-4ef3-b716-2e9bd203c71b/volumes/kubernetes.io~projected/kube-api-access-ws78k DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c94082a0-1953-42aa-9615-7820c50317eb/volumes/kubernetes.io~projected/kube-api-access-9hrxk DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/484e317be04b0101c51b5cba18be2532515aee2a1273866bc3244d2dd3f80cf6/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffd4c9fdf366c90eedd0e4e4dc3a900b1415da494cb5eb5900fd7284e5dcce07/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8aac5b5b10f861462f3bce8dedcefe4a32ec06f4f18b37f7dcc9f531b11a209/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7/volumes/kubernetes.io~projected/kube-api-access-xz78w DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8ebe740aee9ba42efd99eae9ba0b0200c9fe51b2e4cfe1a7c7979c8fb1ac03d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d690421ce580f97edeaafacc976fd731fe38e3a74f58c75b23bf0655bba72dc9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/volumes/kubernetes.io~projected/kube-api-access-gghqz DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4af1cbc86d3089b938ef0df87234ce2a1d3e4b5d2316a8c0c7a02def084697b6/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d25eaef0-b0ac-4326-925b-aaef13840661/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5980f4c704ea616cf3ce505bf5a63811dcedb155d77a1800ccc2a105ca19b1f7/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/971a0263-ee9a-4280-aa61-d385483ca8f0/volumes/kubernetes.io~projected/kube-api-access-76wk9 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8207c74512cebb590a0083a3b94d4c8c3b4694c2d832d078583a4fe158c7a7bf/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3ecd4ba6-46cc-4f76-8984-492c35ad4f40/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1018915fa064c96 MacAddress:96:cb:85:b4:c6:f9 Speed:10000 Mtu:8901} {Name:38470b6e56f6957 MacAddress:86:d5:3b:5a:3a:98 Speed:10000 Mtu:8901} {Name:454ac2a3ad86f32 MacAddress:7a:d8:d5:97:6f:07 Speed:10000 Mtu:8901} {Name:58b38b8f37fbebc MacAddress:4a:16:2b:bc:84:14 Speed:10000 Mtu:8901} {Name:5980f4c704ea616 MacAddress:72:50:63:1c:bb:d6 Speed:10000 Mtu:8901} {Name:7461b2d6073283c MacAddress:8e:92:26:02:2b:01 Speed:10000 Mtu:8901} {Name:8207c74512cebb5 MacAddress:82:89:a1:d8:77:a0 Speed:10000 Mtu:8901} {Name:84f8d19e6f1a828 MacAddress:2a:d7:62:9f:80:e0 Speed:10000 Mtu:8901} {Name:96baecf9a45445c MacAddress:62:63:24:a3:46:bd Speed:10000 Mtu:8901} {Name:a8aac5b5b10f861 MacAddress:d2:12:55:27:92:52 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3d:ef:3c:01:63 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:df:e8:a2:dc:48 Speed:0 Mtu:8901} {Name:d690421ce580f97 MacAddress:e6:03:03:73:3b:dd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3d:ef:3c:01:63 Speed:0 Mtu:9001} {Name:fcf7b919c1218c3 MacAddress:d6:da:5d:3c:4f:94 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c6:5f:b6:68:54:4b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:e0:a2:15:cd:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062286 4868 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062326 4868 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062561 4868 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062577 4868 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-74.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062687 4868 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062695 4868 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062708 4868 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062720 4868 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062737 4868 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062868 4868 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.062968 4868 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063003 4868 kubelet.go:493] "Attempting to sync node with API server" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063012 4868 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063024 4868 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063033 4868 kubelet.go:405] "Adding apiserver pod source" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063041 4868 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063905 4868 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 02:47:38.064038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.063919 4868 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 02:47:38.065013 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.064394 4868 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 02:47:38.066621 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.066603 4868 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 02:47:38.066621 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.066618 4868 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 02:47:38.066758 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.066736 4868 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 02:47:38.066948 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.066932 4868 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 02:47:38.067025 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.066973 4868 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 02:47:38.067089 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067076 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 02:47:38.067146 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067098 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 02:47:38.067146 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067107 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 02:47:38.067146 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067116 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 02:47:38.067146 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067139 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067149 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067158 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067169 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067179 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067188 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067213 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067229 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067267 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 02:47:38.067335 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.067276 4868 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 02:47:38.070756 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.070740 4868 server.go:1263] "Started kubelet" Nov 02 02:47:38.071442 ip-10-0-137-74 systemd[1]: Started Kubernetes Kubelet. Nov 02 02:47:38.071749 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.070804 4868 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 02:47:38.074226 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.072629 4868 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 02:47:38.074226 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.072700 4868 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 02:47:38.074226 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.072972 4868 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 02:47:38.074226 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.073183 4868 server.go:317] "Adding debug handlers to kubelet server" Nov 02 02:47:38.076078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076056 4868 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 02:47:38.076078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076067 4868 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 02:47:38.076568 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076135 4868 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 02:41:58 +0000 UTC" deadline="2028-06-11 22:37:56.428934364 +0000 UTC" Nov 02 02:47:38.076568 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076158 4868 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14107h50m18.352779734s" Nov 02 02:47:38.076568 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076158 4868 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 02:47:38.076568 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.076175 4868 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:47:38.076568 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076187 4868 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 02:47:38.076568 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.076181 4868 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 02:47:38.077392 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.077192 4868 factory.go:55] Registering systemd factory Nov 02 02:47:38.077392 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.077216 4868 factory.go:223] Registration of the systemd container factory successfully Nov 02 02:47:38.078227 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.078207 4868 factory.go:153] Registering CRI-O factory Nov 02 02:47:38.078227 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.078226 4868 factory.go:223] Registration of the crio container factory successfully Nov 02 02:47:38.078416 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.078281 4868 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 02:47:38.078416 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.078298 4868 factory.go:103] Registering Raw factory Nov 02 02:47:38.078416 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.078314 4868 manager.go:1199] Started watching for new ooms in manager Nov 02 02:47:38.078665 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.078650 4868 manager.go:319] Starting recovery of all containers Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079258 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079310 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b96028-853b-4750-a2b3-09658c06aa2e" volumeName="kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-service-ca-bundle" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079325 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a3ee91-5a11-4a4d-8b6d-32273c3c26b5" volumeName="kubernetes.io/secret/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079338 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a0263-ee9a-4280-aa61-d385483ca8f0" volumeName="kubernetes.io/projected/971a0263-ee9a-4280-aa61-d385483ca8f0-kube-api-access-76wk9" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079373 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a0263-ee9a-4280-aa61-d385483ca8f0" volumeName="kubernetes.io/secret/971a0263-ee9a-4280-aa61-d385483ca8f0-serving-cert" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079389 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42726530-4a21-425e-8598-05227546e93f" volumeName="kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-multus-daemon-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079402 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ba41133-2ab9-43ac-94d9-adfade07fee6" volumeName="kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-tmp" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079413 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3744822-b0ed-4aeb-9ab9-f5401a3905f0" volumeName="kubernetes.io/projected/e3744822-b0ed-4aeb-9ab9-f5401a3905f0-kube-api-access-d9xfh" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079426 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a3ee91-5a11-4a4d-8b6d-32273c3c26b5" volumeName="kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-env-overrides" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079436 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b96028-853b-4750-a2b3-09658c06aa2e" volumeName="kubernetes.io/projected/17b96028-853b-4750-a2b3-09658c06aa2e-kube-api-access-nhnfb" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079446 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17e048e5-425c-43fe-8453-f4e30a51df0a" volumeName="kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079458 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17e048e5-425c-43fe-8453-f4e30a51df0a" volumeName="kubernetes.io/projected/17e048e5-425c-43fe-8453-f4e30a51df0a-kube-api-access-9kgmn" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079473 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91547ba7-4770-461b-a34b-0f3a46bdde43" volumeName="kubernetes.io/projected/91547ba7-4770-461b-a34b-0f3a46bdde43-kube-api-access-xvhhr" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079484 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079500 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079510 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bf0a3b-df4f-458d-b0ae-72483d95f2d7" volumeName="kubernetes.io/empty-dir/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-tmp-dir" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079522 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="201c03d0-ed5b-4ccb-b4c2-534fe47511c4" volumeName="kubernetes.io/configmap/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-telemetry-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079534 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d03c81-d243-4970-9334-023b1af99f91" volumeName="kubernetes.io/secret/50d03c81-d243-4970-9334-023b1af99f91-cert" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079546 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0" volumeName="kubernetes.io/projected/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-kube-api-access-gghqz" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079556 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b96028-853b-4750-a2b3-09658c06aa2e" volumeName="kubernetes.io/secret/17b96028-853b-4750-a2b3-09658c06aa2e-serving-cert" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079568 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ee35072-3ed9-4c17-9d67-714aec085b58" volumeName="kubernetes.io/configmap/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079579 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb2c8c59-10f2-4f93-a4bd-1b37406fec26" volumeName="kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079595 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d60a499f-81af-466f-9be9-a581feee8523" volumeName="kubernetes.io/empty-dir/d60a499f-81af-466f-9be9-a581feee8523-data-volume" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079607 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7bf0a3b-df4f-458d-b0ae-72483d95f2d7" volumeName="kubernetes.io/projected/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-kube-api-access-xz78w" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079619 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41d1c58c-4c19-4212-934c-ecaf59792f37" volumeName="kubernetes.io/configmap/41d1c58c-4c19-4212-934c-ecaf59792f37-iptables-alerter-script" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079633 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d03c81-d243-4970-9334-023b1af99f91" volumeName="kubernetes.io/projected/50d03c81-d243-4970-9334-023b1af99f91-kube-api-access-tfgfj" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079644 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079657 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e09d2f-2971-476b-afd8-b3b37be1437b" volumeName="kubernetes.io/projected/60e09d2f-2971-476b-afd8-b3b37be1437b-kube-api-access-6jb6m" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079672 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c94082a0-1953-42aa-9615-7820c50317eb" volumeName="kubernetes.io/secret/c94082a0-1953-42aa-9615-7820c50317eb-serving-cert" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079679 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25eaef0-b0ac-4326-925b-aaef13840661" volumeName="kubernetes.io/projected/d25eaef0-b0ac-4326-925b-aaef13840661-kube-api-access-vt6hz" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079702 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ba41133-2ab9-43ac-94d9-adfade07fee6" volumeName="kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-tuned" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079710 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="201c03d0-ed5b-4ccb-b4c2-534fe47511c4" volumeName="kubernetes.io/projected/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-kube-api-access-js4gq" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079718 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b96028-853b-4750-a2b3-09658c06aa2e" volumeName="kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-trusted-ca-bundle" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079726 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41d1c58c-4c19-4212-934c-ecaf59792f37" volumeName="kubernetes.io/projected/41d1c58c-4c19-4212-934c-ecaf59792f37-kube-api-access-6hnsj" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079733 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42726530-4a21-425e-8598-05227546e93f" volumeName="kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-cni-binary-copy" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079742 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079758 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079766 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c94082a0-1953-42aa-9615-7820c50317eb" volumeName="kubernetes.io/configmap/c94082a0-1953-42aa-9615-7820c50317eb-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079773 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079780 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079787 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b96028-853b-4750-a2b3-09658c06aa2e" volumeName="kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-snapshots" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079799 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a3ee91-5a11-4a4d-8b6d-32273c3c26b5" volumeName="kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079808 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a3ee91-5a11-4a4d-8b6d-32273c3c26b5" volumeName="kubernetes.io/projected/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-kube-api-access-d67nx" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079815 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c9cf6d2-1802-43d0-99ef-2480983f9f52" volumeName="kubernetes.io/projected/4c9cf6d2-1802-43d0-99ef-2480983f9f52-kube-api-access-zqlzv" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079825 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079832 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="872a3ac5-d06b-486c-a18f-99b640928ab3" volumeName="kubernetes.io/projected/872a3ac5-d06b-486c-a18f-99b640928ab3-kube-api-access-ck6t8" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079839 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b96028-853b-4750-a2b3-09658c06aa2e" volumeName="kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-tmp" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079849 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a3ee91-5a11-4a4d-8b6d-32273c3c26b5" volumeName="kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-script-lib" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079857 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079864 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ba41133-2ab9-43ac-94d9-adfade07fee6" volumeName="kubernetes.io/projected/9ba41133-2ab9-43ac-94d9-adfade07fee6-kube-api-access-5bd78" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079871 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25eaef0-b0ac-4326-925b-aaef13840661" volumeName="kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-default-certificate" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079878 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="210af9a6-abe4-4617-bf11-5ca72b7da8f3" volumeName="kubernetes.io/configmap/210af9a6-abe4-4617-bf11-5ca72b7da8f3-nginx-conf" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079886 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42726530-4a21-425e-8598-05227546e93f" volumeName="kubernetes.io/projected/42726530-4a21-425e-8598-05227546e93f-kube-api-access-bf47p" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079894 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079905 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76685eef-ba60-4361-9b35-c7291f159a8b" volumeName="kubernetes.io/projected/76685eef-ba60-4361-9b35-c7291f159a8b-kube-api-access-c2klh" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079916 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4" volumeName="kubernetes.io/projected/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4-kube-api-access-8tzlj" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079935 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17e048e5-425c-43fe-8453-f4e30a51df0a" volumeName="kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079949 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ecd4ba6-46cc-4f76-8984-492c35ad4f40" volumeName="kubernetes.io/configmap/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-konnectivity-ca" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079960 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079972 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0" volumeName="kubernetes.io/configmap/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079983 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9bee798-853b-4ef3-b716-2e9bd203c71b" volumeName="kubernetes.io/projected/e9bee798-853b-4ef3-b716-2e9bd203c71b-kube-api-access-ws78k" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.079993 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b14c538-49b9-4e01-9bb0-8570712848c0" volumeName="kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080005 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c9cf6d2-1802-43d0-99ef-2480983f9f52" volumeName="kubernetes.io/empty-dir/4c9cf6d2-1802-43d0-99ef-2480983f9f52-tmp-dir" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080016 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91547ba7-4770-461b-a34b-0f3a46bdde43" volumeName="kubernetes.io/configmap/91547ba7-4770-461b-a34b-0f3a46bdde43-serviceca" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080026 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d57bc0b-e4b2-401a-a41f-49438f0edb58" volumeName="kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080039 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17e048e5-425c-43fe-8453-f4e30a51df0a" volumeName="kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-binary-copy" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080050 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c9cf6d2-1802-43d0-99ef-2480983f9f52" volumeName="kubernetes.io/configmap/4c9cf6d2-1802-43d0-99ef-2480983f9f52-config-volume" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080061 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ee35072-3ed9-4c17-9d67-714aec085b58" volumeName="kubernetes.io/projected/4ee35072-3ed9-4c17-9d67-714aec085b58-kube-api-access-d77r8" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080072 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a0263-ee9a-4280-aa61-d385483ca8f0" volumeName="kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-config" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080083 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b855c6cb-d565-4687-bd13-248d85111459" volumeName="kubernetes.io/projected/b855c6cb-d565-4687-bd13-248d85111459-kube-api-access-grznx" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080095 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c94082a0-1953-42aa-9615-7820c50317eb" volumeName="kubernetes.io/projected/c94082a0-1953-42aa-9615-7820c50317eb-kube-api-access-9hrxk" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080106 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25eaef0-b0ac-4326-925b-aaef13840661" volumeName="kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-stats-auth" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080117 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c" volumeName="kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080129 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ecd4ba6-46cc-4f76-8984-492c35ad4f40" volumeName="kubernetes.io/secret/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-agent-certs" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080140 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a0263-ee9a-4280-aa61-d385483ca8f0" volumeName="kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-trusted-ca" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080151 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0" volumeName="kubernetes.io/secret/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-serving-cert" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080163 4868 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d60a499f-81af-466f-9be9-a581feee8523" volumeName="kubernetes.io/projected/d60a499f-81af-466f-9be9-a581feee8523-kube-api-access-xnr8z" seLinuxMountContext="" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080175 4868 reconstruct.go:97] "Volume reconstruction finished" Nov 02 02:47:38.080378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.080183 4868 reconciler.go:29] "Reconciler: start to sync state" Nov 02 02:47:38.093337 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.093320 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/1.log" Nov 02 02:47:38.093912 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.093894 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/1.log" Nov 02 02:47:38.105431 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.105401 4868 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 02:47:38.106998 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.106978 4868 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 02:47:38.106998 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.106995 4868 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 02:47:38.107127 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.107010 4868 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 02:47:38.107127 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.107052 4868 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 02:47:38.141212 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.141151 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/1.log" Nov 02 02:47:38.141212 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.141186 4868 generic.go:356] "Generic (PLEG): container finished" podID="971a0263-ee9a-4280-aa61-d385483ca8f0" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" exitCode=255 Nov 02 02:47:38.147720 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.147692 4868 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="e53e3d9a48f3174ebd0c42ecf3c7d12725f5832a95f91345cbc54a68ec3a3063" exitCode=0 Nov 02 02:47:38.147720 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.147719 4868 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="a5b6211630fd4ec1be37004716e5a2ec53fc85e7c3a7cf9da262410a2badb29f" exitCode=0 Nov 02 02:47:38.147853 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.147730 4868 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="c9ee9b97411b834561567ab3ceece570cbb3b8dcd3eb8e2541565d9f6a05e063" exitCode=0 Nov 02 02:47:38.147853 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.147742 4868 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="149061d0556b5854570b5a70c0f3ff80b54310811532368d4d2de124b3864187" exitCode=0 Nov 02 02:47:38.147853 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.147752 4868 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="82b46a909150459e4c280bc3a6b395f365d01c5372f91166928764b8d6c108a7" exitCode=0 Nov 02 02:47:38.147853 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.147764 4868 generic.go:356] "Generic (PLEG): container finished" podID="17e048e5-425c-43fe-8453-f4e30a51df0a" containerID="df0ce24cec1139d3599fe0d8a06b9bbd09880cd158d51ede8fb953538cd0d9c5" exitCode=0 Nov 02 02:47:38.152473 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.152443 4868 generic.go:356] "Generic (PLEG): container finished" podID="60aa7d07786267e9b48b181b1bcbc45b" containerID="cb74b861e3a9e8f7428631b6f2c7cf02041e66fbbc6d868e783b197576b45db2" exitCode=0 Nov 02 02:47:38.165058 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.165043 4868 manager.go:324] Recovery completed Nov 02 02:47:38.176347 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.176324 4868 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:47:38.182580 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.182562 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.183591 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.183574 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.183688 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.183609 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.183688 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.183624 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.185642 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185624 4868 cpu_manager.go:226] "Starting" policy="none" Nov 02 02:47:38.185642 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185642 4868 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 02:47:38.185749 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185662 4868 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 02:47:38.185749 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185628 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.185837 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185819 4868 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 02:47:38.185880 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185839 4868 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 02:47:38.185880 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185874 4868 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 02:47:38.185942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185882 4868 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 02:47:38.185942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185890 4868 policy_none.go:50] "Start" Nov 02 02:47:38.185942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185906 4868 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 02:47:38.185942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.185918 4868 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 02:47:38.186112 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.186052 4868 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 02:47:38.186112 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.186063 4868 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 02:47:38.186112 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.186071 4868 policy_none.go:44] "Start" Nov 02 02:47:38.186590 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.186575 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.186641 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.186601 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.186641 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.186611 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.190129 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.190107 4868 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 02:47:38.192751 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.192738 4868 manager.go:341] "Starting Device Plugin manager" Nov 02 02:47:38.192825 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.192765 4868 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 02:47:38.192825 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.192777 4868 server.go:87] "Starting device plugin registration server" Nov 02 02:47:38.193062 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193048 4868 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 02:47:38.193125 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193062 4868 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 02:47:38.193183 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193166 4868 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 02:47:38.193318 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193281 4868 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 02:47:38.193429 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193324 4868 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 02:47:38.193429 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193331 4868 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 02:47:38.193546 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.193389 4868 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T02:47:38.193313491Z","UUID":"ced6f5f2-a3ca-47a2-a211-da6b357a5a5a","Handler":null,"Name":"","Endpoint":""} Nov 02 02:47:38.194638 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.194617 4868 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 02:47:38.194738 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.194657 4868 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" Nov 02 02:47:38.194899 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.194885 4868 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 02:47:38.194983 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.194914 4868 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 02:47:38.207740 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.207276 4868 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal"] Nov 02 02:47:38.207835 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.207822 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.208536 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.208515 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.208602 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.208545 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.208602 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.208555 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.208818 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.208807 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.208950 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.208936 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.211117 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211100 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.211117 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211112 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.211238 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211128 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.211238 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211133 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.211238 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211137 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.211238 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211143 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.211386 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.211258 4868 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.211435 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211409 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"f403bae9d2c2c1c33d0488e61f8d37c6","Type":"ContainerStarted","Data":"fac56e3d1f49e2a1f216d4bf826db52ddfff4adb7cd8d780b93e140cadc778d1"} Nov 02 02:47:38.211472 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211443 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"f403bae9d2c2c1c33d0488e61f8d37c6","Type":"ContainerStarted","Data":"9d31e06b8f9f3582acf72f14c86363cb1d8ea1d475145c5a63e558cf34da938f"} Nov 02 02:47:38.211516 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211507 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"60aa7d07786267e9b48b181b1bcbc45b","Type":"ContainerStarted","Data":"4cbf679b5057c4e04095965b6ec130bd5c1d16242bb75e0c78f324cabadf83c0"} Nov 02 02:47:38.211553 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211522 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"60aa7d07786267e9b48b181b1bcbc45b","Type":"ContainerDied","Data":"cb74b861e3a9e8f7428631b6f2c7cf02041e66fbbc6d868e783b197576b45db2"} Nov 02 02:47:38.211553 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211534 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" event={"ID":"60aa7d07786267e9b48b181b1bcbc45b","Type":"ContainerStarted","Data":"b4c960bc1586afd0e33977cdc536f48d079ae70590e879f49c6951dc1acf6eca"} Nov 02 02:47:38.211553 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.211511 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.212233 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.212219 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.212313 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.212246 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.212313 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.212259 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.212443 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.212427 4868 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-74.us-west-1.compute.internal\" not found" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.220097 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.220068 4868 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:47:38.267746 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.267724 4868 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:47:38.269947 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.269931 4868 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 02:47:38.270034 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.269975 4868 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:47:38.277178 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.277161 4868 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.293317 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.293297 4868 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 02:47:38.294139 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.294125 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 02:47:38.294200 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.294150 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 02:47:38.294200 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.294159 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 02:47:38.294200 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.294193 4868 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.319927 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.319911 4868 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 02:47:38.335199 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.335179 4868 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 02:47:38.335268 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.335226 4868 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.335268 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.335237 4868 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.370233 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.370215 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.370299 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.370249 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.370299 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.370266 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f403bae9d2c2c1c33d0488e61f8d37c6-config\") pod \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"f403bae9d2c2c1c33d0488e61f8d37c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.391985 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.391940 4868 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 02:47:38.391985 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:38.391979 4868 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.418489 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.418471 4868 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.418564 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.418518 4868 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.418881 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.418868 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeNotReady" Nov 02 02:47:38.418932 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.418895 4868 setters.go:546] "Node became not ready" node="ip-10-0-137-74.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T02:47:38Z","lastTransitionTime":"2026-11-02T02:47:38Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 02:47:38.429428 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.429403 4868 nodeinfomanager.go:471] existing CSINode "ip-10-0-137-74.us-west-1.compute.internal" is owned by different node (oldNodeID="0a339dca-77ab-4397-93e0-ab9c5da64edc", newNodeID=""), cleaning up... Nov 02 02:47:38.471159 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.471140 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.471228 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.471164 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.471228 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.471181 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f403bae9d2c2c1c33d0488e61f8d37c6-config\") pod \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"f403bae9d2c2c1c33d0488e61f8d37c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.471228 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.471217 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f403bae9d2c2c1c33d0488e61f8d37c6-config\") pod \"kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"f403bae9d2c2c1c33d0488e61f8d37c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.471316 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.471241 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.471316 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.471251 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60aa7d07786267e9b48b181b1bcbc45b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" (UID: \"60aa7d07786267e9b48b181b1bcbc45b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:38.687510 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.687447 4868 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-74.us-west-1.compute.internal" event="NodeReady" Nov 02 02:47:38.687597 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:38.687526 4868 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 02:47:39.063393 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.063309 4868 apiserver.go:52] "Watching apiserver" Nov 02 02:47:39.159120 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.159098 4868 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:39.215581 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.215558 4868 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 02:47:39.216235 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.216210 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq","openshift-cluster-node-tuning-operator/tuned-76cvn","openshift-ingress-canary/ingress-canary-rgk6v","openshift-network-console/networking-console-plugin-86c478bb88-dn6x6","openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq","openshift-insights/insights-runtime-extractor-5jsrl","kube-system/kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal","kube-system/kas-connection-checker-685895c7c9-q69cv","openshift-multus/network-metrics-daemon-cvzpz","openshift-multus/multus-9kpxq","openshift-console-operator/console-operator-7cd56b8dbc-k6f8b","openshift-multus/multus-additional-cni-plugins-n56m9","openshift-ovn-kubernetes/ovnkube-node-q6wpj","openshift-network-operator/iptables-alerter-zmp76","kube-system/kas-connection-checker-685895c7c9-prxz2","kube-system/kas-connection-checker-685895c7c9-ldnnl","kube-system/konnectivity-agent-v8xt2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425","openshift-image-registry/image-registry-678859df4-mxqvr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l","openshift-dns/dns-default-rvk96","openshift-dns/node-resolver-rfzxg","openshift-network-diagnostics/network-check-target-d42js","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal","openshift-ingress/router-default-648db889-4lftz","openshift-insights/insights-operator-7c4986d449-djgk5","openshift-image-registry/image-registry-7cb8549547-trw7f","kube-system/global-pull-secret-syncer-xhmws","openshift-image-registry/node-ca-5sb5c","openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2"] Nov 02 02:47:39.217564 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.217544 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.217968 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.217938 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.218246 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.218223 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.218562 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.218543 4868 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 02:47:39.218653 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.218609 4868 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-74.us-west-1.compute.internal" Nov 02 02:47:39.219432 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.219415 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.221572 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.221552 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:39.230535 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.230514 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:39.231095 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.231081 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.231611 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.231599 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.232650 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.232632 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.269840 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.269823 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.270125 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.270110 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.271150 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271134 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.271206 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271149 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.271579 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271559 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.271684 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271605 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 02:47:39.271820 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271803 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 02:47:39.271867 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271806 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 02:47:39.271867 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271806 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.271867 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271809 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 02:47:39.271867 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271811 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271913 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271916 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271937 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.271939 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272029 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272048 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272047 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272051 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 02:47:39.272126 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272123 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hws9m\"" Nov 02 02:47:39.272617 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272149 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 02:47:39.272617 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272156 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8tvz9\"" Nov 02 02:47:39.272617 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272344 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6b8nz\"" Nov 02 02:47:39.272617 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272387 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c7bng\"" Nov 02 02:47:39.272617 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272398 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-82426\"" Nov 02 02:47:39.272992 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.272978 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 02:47:39.278735 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.278721 4868 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 02:47:39.318834 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.318783 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.318834 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.318804 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.318834 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.318831 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.319294 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.319282 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.321058 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321044 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.321119 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321057 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 02:47:39.321119 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321097 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.321207 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321132 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.321207 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321144 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.321207 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321152 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 02:47:39.321207 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321162 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.321207 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321173 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.321207 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321206 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 02:47:39.321517 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321214 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.321517 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321307 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 02:47:39.321517 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321364 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 02:47:39.321517 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321492 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 02:47:39.321517 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321510 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 02:47:39.321769 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321644 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dgnd5\"" Nov 02 02:47:39.321998 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.321981 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vbwmz\"" Nov 02 02:47:39.322066 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.322025 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-msd8h\"" Nov 02 02:47:39.322325 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.322308 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 02:47:39.322416 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.322400 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gvcps\"" Nov 02 02:47:39.323040 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.323022 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 02:47:39.323097 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.323060 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 02:47:39.369802 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.369783 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 02:47:39.370038 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370022 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 02:47:39.370102 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370030 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370102 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370054 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370102 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370062 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370253 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370200 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 02:47:39.370253 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370210 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370253 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370211 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370253 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370226 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 02:47:39.370471 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370293 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 02:47:39.370471 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370334 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370471 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370380 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370493 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370517 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370528 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370530 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370557 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370581 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370589 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370788 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370801 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370805 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370804 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370829 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370891 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370908 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 02:47:39.370956 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.370896 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 02:47:39.371731 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.371705 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 02:47:39.372165 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372146 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 02:47:39.372439 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372419 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 02:47:39.372549 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372466 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 02:47:39.372614 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372572 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gdqg4\"" Nov 02 02:47:39.372760 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372705 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 02:47:39.372760 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372706 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 02:47:39.372881 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372815 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6md5s\"" Nov 02 02:47:39.372928 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.372902 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvbct\"" Nov 02 02:47:39.374773 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.373090 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 02:47:39.374773 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.373208 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 02:47:39.375136 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375115 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6ddg2\"" Nov 02 02:47:39.375194 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375140 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 02:47:39.375194 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375163 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b8q6l\"" Nov 02 02:47:39.375194 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375171 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wsjnf\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375195 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2g54f\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375200 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5mjxc\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375201 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cjmh9\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375223 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x92pp\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375235 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pbbl9\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375251 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375238 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5v9mr\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375270 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7jkxs\"" Nov 02 02:47:39.375305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375282 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nvmsr\"" Nov 02 02:47:39.375579 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375340 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gkth\"" Nov 02 02:47:39.375579 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375410 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cgnbb\"" Nov 02 02:47:39.375579 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375511 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 02:47:39.375729 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.375708 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ghkrt\"" Nov 02 02:47:39.376346 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376329 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-default-certificate\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.376407 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376372 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-node-log\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.376407 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376398 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77r8\" (UniqueName: \"kubernetes.io/projected/4ee35072-3ed9-4c17-9d67-714aec085b58-kube-api-access-d77r8\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.376466 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376415 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-tmp\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.376466 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376433 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grznx\" (UniqueName: \"kubernetes.io/projected/b855c6cb-d565-4687-bd13-248d85111459-kube-api-access-grznx\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:39.376466 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376448 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.376584 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376551 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-tmp\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.376632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376623 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.376679 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376622 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-default-certificate\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.376679 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376651 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgfj\" (UniqueName: \"kubernetes.io/projected/50d03c81-d243-4970-9334-023b1af99f91-kube-api-access-tfgfj\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:39.376775 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376680 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.376775 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376707 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.376775 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376727 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.376775 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376755 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67nx\" (UniqueName: \"kubernetes.io/projected/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-kube-api-access-d67nx\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.376893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376794 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlzv\" (UniqueName: \"kubernetes.io/projected/4c9cf6d2-1802-43d0-99ef-2480983f9f52-kube-api-access-zqlzv\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.376893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376808 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.376893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376849 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.376893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376867 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-bin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.376893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376883 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17b96028-853b-4750-a2b3-09658c06aa2e-serving-cert\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.377078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376900 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghqz\" (UniqueName: \"kubernetes.io/projected/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-kube-api-access-gghqz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:39.377078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376921 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.377078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.376954 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-hosts-file\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.377078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377012 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-tmp-dir\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377090 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17b96028-853b-4750-a2b3-09658c06aa2e-serving-cert\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377114 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-multus\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377136 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-hostroot\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377159 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf47p\" (UniqueName: \"kubernetes.io/projected/42726530-4a21-425e-8598-05227546e93f-kube-api-access-bf47p\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377179 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-tmp-dir\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377195 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-config\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.377231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377223 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzlj\" (UniqueName: \"kubernetes.io/projected/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4-kube-api-access-8tzlj\") pod \"kas-connection-checker-685895c7c9-ldnnl\" (UID: \"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4\") " pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377259 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrxk\" (UniqueName: \"kubernetes.io/projected/c94082a0-1953-42aa-9615-7820c50317eb-kube-api-access-9hrxk\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377284 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-config\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377306 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971a0263-ee9a-4280-aa61-d385483ca8f0-serving-cert\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377329 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-multus-daemon-config\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377337 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377368 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.377509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377465 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-config\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377592 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377607 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/971a0263-ee9a-4280-aa61-d385483ca8f0-serving-cert\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377620 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-config\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377626 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-multus-daemon-config\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377628 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377663 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60a499f-81af-466f-9be9-a581feee8523-crio-socket\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.377732 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377693 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377734 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377771 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhhr\" (UniqueName: \"kubernetes.io/projected/91547ba7-4770-461b-a34b-0f3a46bdde43-kube-api-access-xvhhr\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377826 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-trusted-ca\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377869 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-snapshots\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377904 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377906 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-tuned\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377928 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377949 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-ovn\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377975 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-tuned\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377979 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-snapshots\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.377976 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378012 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:39.378061 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378057 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41d1c58c-4c19-4212-934c-ecaf59792f37-iptables-alerter-script\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378078 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws78k\" (UniqueName: \"kubernetes.io/projected/e9bee798-853b-4ef3-b716-2e9bd203c71b-kube-api-access-ws78k\") pod \"kas-connection-checker-685895c7c9-q69cv\" (UID: \"e9bee798-853b-4ef3-b716-2e9bd203c71b\") " pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378104 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnfb\" (UniqueName: \"kubernetes.io/projected/17b96028-853b-4750-a2b3-09658c06aa2e-kube-api-access-nhnfb\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378121 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378141 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-kubelet\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378084 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/971a0263-ee9a-4280-aa61-d385483ca8f0-trusted-ca\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378165 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-tmp\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378191 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378214 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9cf6d2-1802-43d0-99ef-2480983f9f52-config-volume\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378241 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17b96028-853b-4750-a2b3-09658c06aa2e-tmp\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378250 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378244 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-k8s-cni-cncf-io\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378286 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41d1c58c-4c19-4212-934c-ecaf59792f37-host-slash\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378306 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41d1c58c-4c19-4212-934c-ecaf59792f37-iptables-alerter-script\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378314 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/210af9a6-abe4-4617-bf11-5ca72b7da8f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378380 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378384 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378429 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378430 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378451 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9cf6d2-1802-43d0-99ef-2480983f9f52-config-volume\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378458 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378514 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378527 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/210af9a6-abe4-4617-bf11-5ca72b7da8f3-nginx-conf\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378547 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6hz\" (UniqueName: \"kubernetes.io/projected/d25eaef0-b0ac-4326-925b-aaef13840661-kube-api-access-vt6hz\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.378632 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378608 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378675 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6t8\" (UniqueName: \"kubernetes.io/projected/872a3ac5-d06b-486c-a18f-99b640928ab3-kube-api-access-ck6t8\") pod \"network-check-source-6f78d59f99-z89m4\" (UID: \"872a3ac5-d06b-486c-a18f-99b640928ab3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378723 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-socket-dir-parent\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378832 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-socket-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378875 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378894 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-netns\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378909 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-etc-kubernetes\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378925 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-konnectivity-ca\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.378960 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jb6m\" (UniqueName: \"kubernetes.io/projected/60e09d2f-2971-476b-afd8-b3b37be1437b-kube-api-access-6jb6m\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379073 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91547ba7-4770-461b-a34b-0f3a46bdde43-host\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379104 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379111 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-original-pull-secret\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379134 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnr8z\" (UniqueName: \"kubernetes.io/projected/d60a499f-81af-466f-9be9-a581feee8523-kube-api-access-xnr8z\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379175 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c94082a0-1953-42aa-9615-7820c50317eb-config\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379204 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-konnectivity-ca\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379227 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9cf6d2-1802-43d0-99ef-2480983f9f52-tmp-dir\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379309 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379335 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c94082a0-1953-42aa-9615-7820c50317eb-config\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379339 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9cf6d2-1802-43d0-99ef-2480983f9f52-tmp-dir\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379338 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379391 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379410 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379415 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-systemd\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379435 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-registration-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379538 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-netns\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379551 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-cni-binary-copy\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379571 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379599 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379628 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-device-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379660 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-sys\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379692 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-host\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379710 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.379728 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379725 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bd78\" (UniqueName: \"kubernetes.io/projected/9ba41133-2ab9-43ac-94d9-adfade07fee6-kube-api-access-5bd78\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379757 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379773 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-etc-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379796 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-netd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379871 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/e3744822-b0ed-4aeb-9ab9-f5401a3905f0-kube-api-access-d9xfh\") pod \"volume-data-source-validator-58c9f5f46d-7l425\" (UID: \"e3744822-b0ed-4aeb-9ab9-f5401a3905f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379896 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-agent-certs\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.379984 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380002 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380020 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4gq\" (UniqueName: \"kubernetes.io/projected/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-kube-api-access-js4gq\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380171 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ecd4ba6-46cc-4f76-8984-492c35ad4f40-agent-certs\") pod \"konnectivity-agent-v8xt2\" (UID: \"3ecd4ba6-46cc-4f76-8984-492c35ad4f40\") " pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380200 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2klh\" (UniqueName: \"kubernetes.io/projected/76685eef-ba60-4361-9b35-c7291f159a8b-kube-api-access-c2klh\") pod \"kas-connection-checker-685895c7c9-prxz2\" (UID: \"76685eef-ba60-4361-9b35-c7291f159a8b\") " pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380233 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-service-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380239 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17e048e5-425c-43fe-8453-f4e30a51df0a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380260 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-kubernetes\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380286 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380311 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-conf-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380346 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysconfig\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380378 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b96028-853b-4750-a2b3-09658c06aa2e-service-ca-bundle\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380406 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380444 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-system-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380473 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-kubelet\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380504 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnsj\" (UniqueName: \"kubernetes.io/projected/41d1c58c-4c19-4212-934c-ecaf59792f37-kube-api-access-6hnsj\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380533 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380556 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380581 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-kubelet-config\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380724 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-cnibin\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380742 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-cnibin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380758 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovn-node-metrics-cert\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380773 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-sys-fs\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380786 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380823 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-lib-modules\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380898 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380918 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-stats-auth\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380917 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380936 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-var-lib-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380952 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-dbus\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380956 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovn-node-metrics-cert\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380966 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.380995 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-cni-binary-copy\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381015 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381057 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.381051 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381083 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-bin\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381104 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381120 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-stats-auth\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381161 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381173 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42726530-4a21-425e-8598-05227546e93f-cni-binary-copy\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381186 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381201 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-systemd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381331 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c94082a0-1953-42aa-9615-7820c50317eb-serving-cert\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381342 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381390 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-system-cni-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381439 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgmn\" (UniqueName: \"kubernetes.io/projected/17e048e5-425c-43fe-8453-f4e30a51df0a-kube-api-access-9kgmn\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381464 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381486 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-var-lib-kubelet\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381499 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c94082a0-1953-42aa-9615-7820c50317eb-serving-cert\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381509 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-slash\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381549 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-multus-certs\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381591 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91547ba7-4770-461b-a34b-0f3a46bdde43-serviceca\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381653 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381689 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d03c81-d243-4970-9334-023b1af99f91-cert\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381712 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz78w\" (UniqueName: \"kubernetes.io/projected/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-kube-api-access-xz78w\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381736 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-systemd-units\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381758 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-log-socket\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381781 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-env-overrides\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381810 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-modprobe-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381828 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-conf\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381849 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-script-lib\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381853 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91547ba7-4770-461b-a34b-0f3a46bdde43-serviceca\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381891 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wk9\" (UniqueName: \"kubernetes.io/projected/971a0263-ee9a-4280-aa61-d385483ca8f0-kube-api-access-76wk9\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381925 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381952 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-run\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.381977 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60a499f-81af-466f-9be9-a581feee8523-data-volume\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.382072 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-ovnkube-script-lib\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.382100 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d60a499f-81af-466f-9be9-a581feee8523-data-volume\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.382189 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.382199 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-env-overrides\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.382275 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d03c81-d243-4970-9334-023b1af99f91-cert\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:39.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.382340 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.434193 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.434167 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grznx\" (UniqueName: \"kubernetes.io/projected/b855c6cb-d565-4687-bd13-248d85111459-kube-api-access-grznx\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:39.434662 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.434643 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzlj\" (UniqueName: \"kubernetes.io/projected/d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4-kube-api-access-8tzlj\") pod \"kas-connection-checker-685895c7c9-ldnnl\" (UID: \"d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4\") " pod="kube-system/kas-connection-checker-685895c7c9-ldnnl" Nov 02 02:47:39.434808 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.434791 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.435958 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.435941 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77r8\" (UniqueName: \"kubernetes.io/projected/4ee35072-3ed9-4c17-9d67-714aec085b58-kube-api-access-d77r8\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.436231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.436212 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bd78\" (UniqueName: \"kubernetes.io/projected/9ba41133-2ab9-43ac-94d9-adfade07fee6-kube-api-access-5bd78\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.436540 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.436521 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf47p\" (UniqueName: \"kubernetes.io/projected/42726530-4a21-425e-8598-05227546e93f-kube-api-access-bf47p\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.436605 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.436582 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6t8\" (UniqueName: \"kubernetes.io/projected/872a3ac5-d06b-486c-a18f-99b640928ab3-kube-api-access-ck6t8\") pod \"network-check-source-6f78d59f99-z89m4\" (UID: \"872a3ac5-d06b-486c-a18f-99b640928ab3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z89m4" Nov 02 02:47:39.436605 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.436591 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgfj\" (UniqueName: \"kubernetes.io/projected/50d03c81-d243-4970-9334-023b1af99f91-kube-api-access-tfgfj\") pod \"ingress-canary-rgk6v\" (UID: \"50d03c81-d243-4970-9334-023b1af99f91\") " pod="openshift-ingress-canary/ingress-canary-rgk6v" Nov 02 02:47:39.437019 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.436999 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnr8z\" (UniqueName: \"kubernetes.io/projected/d60a499f-81af-466f-9be9-a581feee8523-kube-api-access-xnr8z\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.437095 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437027 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghqz\" (UniqueName: \"kubernetes.io/projected/ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0-kube-api-access-gghqz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2\" (UID: \"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" Nov 02 02:47:39.437177 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437156 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnfb\" (UniqueName: \"kubernetes.io/projected/17b96028-853b-4750-a2b3-09658c06aa2e-kube-api-access-nhnfb\") pod \"insights-operator-7c4986d449-djgk5\" (UID: \"17b96028-853b-4750-a2b3-09658c06aa2e\") " pod="openshift-insights/insights-operator-7c4986d449-djgk5" Nov 02 02:47:39.437229 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437184 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlzv\" (UniqueName: \"kubernetes.io/projected/4c9cf6d2-1802-43d0-99ef-2480983f9f52-kube-api-access-zqlzv\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.437229 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437197 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67nx\" (UniqueName: \"kubernetes.io/projected/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-kube-api-access-d67nx\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.437747 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437722 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgmn\" (UniqueName: \"kubernetes.io/projected/17e048e5-425c-43fe-8453-f4e30a51df0a-kube-api-access-9kgmn\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.437869 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437808 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhhr\" (UniqueName: \"kubernetes.io/projected/91547ba7-4770-461b-a34b-0f3a46bdde43-kube-api-access-xvhhr\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.437869 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437821 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrxk\" (UniqueName: \"kubernetes.io/projected/c94082a0-1953-42aa-9615-7820c50317eb-kube-api-access-9hrxk\") pod \"service-ca-operator-544d749c8c-9vhbq\" (UID: \"c94082a0-1953-42aa-9615-7820c50317eb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" Nov 02 02:47:39.437936 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.437883 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.438394 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.438378 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2klh\" (UniqueName: \"kubernetes.io/projected/76685eef-ba60-4361-9b35-c7291f159a8b-kube-api-access-c2klh\") pod \"kas-connection-checker-685895c7c9-prxz2\" (UID: \"76685eef-ba60-4361-9b35-c7291f159a8b\") " pod="kube-system/kas-connection-checker-685895c7c9-prxz2" Nov 02 02:47:39.438528 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.438512 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/e3744822-b0ed-4aeb-9ab9-f5401a3905f0-kube-api-access-d9xfh\") pod \"volume-data-source-validator-58c9f5f46d-7l425\" (UID: \"e3744822-b0ed-4aeb-9ab9-f5401a3905f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7l425" Nov 02 02:47:39.438811 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.438792 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws78k\" (UniqueName: \"kubernetes.io/projected/e9bee798-853b-4ef3-b716-2e9bd203c71b-kube-api-access-ws78k\") pod \"kas-connection-checker-685895c7c9-q69cv\" (UID: \"e9bee798-853b-4ef3-b716-2e9bd203c71b\") " pod="kube-system/kas-connection-checker-685895c7c9-q69cv" Nov 02 02:47:39.439251 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439234 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.439344 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439284 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.439468 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439441 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4gq\" (UniqueName: \"kubernetes.io/projected/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-kube-api-access-js4gq\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.439468 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439462 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dmj\" (UniqueName: \"kubernetes.io/projected/bb2c8c59-10f2-4f93-a4bd-1b37406fec26-kube-api-access-k6dmj\") pod \"network-check-target-d42js\" (UID: \"bb2c8c59-10f2-4f93-a4bd-1b37406fec26\") " pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:39.439579 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439480 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jb6m\" (UniqueName: \"kubernetes.io/projected/60e09d2f-2971-476b-afd8-b3b37be1437b-kube-api-access-6jb6m\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.439670 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439654 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz78w\" (UniqueName: \"kubernetes.io/projected/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-kube-api-access-xz78w\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.439722 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439708 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnsj\" (UniqueName: \"kubernetes.io/projected/41d1c58c-4c19-4212-934c-ecaf59792f37-kube-api-access-6hnsj\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.439753 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.439722 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6hz\" (UniqueName: \"kubernetes.io/projected/d25eaef0-b0ac-4326-925b-aaef13840661-kube-api-access-vt6hz\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.440082 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.440067 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wk9\" (UniqueName: \"kubernetes.io/projected/971a0263-ee9a-4280-aa61-d385483ca8f0-kube-api-access-76wk9\") pod \"console-operator-7cd56b8dbc-k6f8b\" (UID: \"971a0263-ee9a-4280-aa61-d385483ca8f0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:39.482721 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482701 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-kubernetes\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.482783 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482727 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:39.482783 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482743 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-conf-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.482783 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482759 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysconfig\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482782 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482805 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-system-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482824 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-kubernetes\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482828 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-kubelet\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482844 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-conf-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482862 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482864 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482867 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-kubelet\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482827 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysconfig\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482870 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-system-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.482895 4868 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:39.482921 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482921 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-kubelet-config\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.482911 4868 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.482896 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-kubelet-config\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.482975 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.982955828 +0000 UTC m=+1.978656765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483016 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.983006376 +0000 UTC m=+1.978707298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : configmap "kube-rbac-proxy" not found Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483035 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-cnibin\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483063 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-cnibin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483076 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-cnibin\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483087 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-sys-fs\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483111 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483133 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-lib-modules\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483135 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-cnibin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483160 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-var-lib-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483133 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-sys-fs\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483182 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-dbus\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483215 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-var-lib-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483241 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c-dbus\") pod \"global-pull-secret-syncer-xhmws\" (UID: \"f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c\") " pod="kube-system/global-pull-secret-syncer-xhmws" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483247 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-lib-modules\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483217 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483178 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-cni-dir\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483283 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-bin\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483300 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483315 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-systemd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483330 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-system-cni-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483344 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483346 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-bin\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483258 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483389 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-systemd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483397 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17e048e5-425c-43fe-8453-f4e30a51df0a-system-cni-dir\") pod \"multus-additional-cni-plugins-n56m9\" (UID: \"17e048e5-425c-43fe-8453-f4e30a51df0a\") " pod="openshift-multus/multus-additional-cni-plugins-n56m9" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483401 4868 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483422 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-var-lib-kubelet\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483443 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.983430081 +0000 UTC m=+1.979131041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483458 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-var-lib-kubelet\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483465 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-slash\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483492 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-multus-certs\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.483476 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483445 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483519 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-systemd-units\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483533 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-slash\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483541 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-multus-certs\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483542 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-log-socket\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483563 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-systemd-units\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483571 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-modprobe-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483583 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-log-socket\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483585 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-conf\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483619 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-run\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483631 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-modprobe-d\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483648 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-node-log\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483662 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-sysctl-conf\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483671 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-run\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483683 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483691 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-node-log\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483709 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483724 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483734 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483736 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-bin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483756 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483765 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483775 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.983765145 +0000 UTC m=+1.979466092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483786 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-bin\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483798 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-hosts-file\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483814 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-multus\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483828 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-hostroot\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.483840 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.983826437 +0000 UTC m=+1.979527381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483847 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-hostroot\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483842 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7bf0a3b-df4f-458d-b0ae-72483d95f2d7-hosts-file\") pod \"node-resolver-rfzxg\" (UID: \"f7bf0a3b-df4f-458d-b0ae-72483d95f2d7\") " pod="openshift-dns/node-resolver-rfzxg" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483860 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-var-lib-cni-multus\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483870 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483897 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60a499f-81af-466f-9be9-a581feee8523-crio-socket\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483919 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483947 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-ovn\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483956 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d60a499f-81af-466f-9be9-a581feee8523-crio-socket\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483948 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.483982 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-run-ovn\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484016 4868 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484048 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.984038534 +0000 UTC m=+1.979739455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484062 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484080 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-kubelet\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484096 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-k8s-cni-cncf-io\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484106 4868 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484125 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41d1c58c-4c19-4212-934c-ecaf59792f37-host-slash\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484109 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41d1c58c-4c19-4212-934c-ecaf59792f37-host-slash\") pod \"iptables-alerter-zmp76\" (UID: \"41d1c58c-4c19-4212-934c-ecaf59792f37\") " pod="openshift-network-operator/iptables-alerter-zmp76" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484152 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.984139478 +0000 UTC m=+1.979840425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : secret "metrics-daemon-secret" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484158 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-kubelet\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484181 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484197 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-k8s-cni-cncf-io\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484208 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484231 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-socket-dir-parent\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484246 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-socket-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484270 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-netns\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484292 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-etc-kubernetes\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484298 4868 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484315 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91547ba7-4770-461b-a34b-0f3a46bdde43-host\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484326 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.984317132 +0000 UTC m=+1.980018052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484322 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484369 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-etc-kubernetes\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484376 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-multus-socket-dir-parent\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484371 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42726530-4a21-425e-8598-05227546e93f-host-run-netns\") pod \"multus-9kpxq\" (UID: \"42726530-4a21-425e-8598-05227546e93f\") " pod="openshift-multus/multus-9kpxq" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484387 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484377 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-socket-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484413 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91547ba7-4770-461b-a34b-0f3a46bdde43-host\") pod \"node-ca-5sb5c\" (UID: \"91547ba7-4770-461b-a34b-0f3a46bdde43\") " pod="openshift-image-registry/node-ca-5sb5c" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484419 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-systemd\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484449 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484453 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-etc-systemd\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484461 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484499 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-registration-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484509 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.984495547 +0000 UTC m=+1.980196487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484455 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-registration-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484533 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-netns\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484551 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484565 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-device-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484579 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-sys\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484590 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-host\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484605 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484623 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/60e09d2f-2971-476b-afd8-b3b37be1437b-device-dir\") pod \"aws-ebs-csi-driver-node-8rv6l\" (UID: \"60e09d2f-2971-476b-afd8-b3b37be1437b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rv6l" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484623 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484633 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-sys\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484624 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-etc-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484649 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-run-netns\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484656 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-etc-openvswitch\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484679 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-netd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484684 4868 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484700 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484723 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39a3ee91-5a11-4a4d-8b6d-32273c3c26b5-host-cni-netd\") pod \"ovnkube-node-q6wpj\" (UID: \"39a3ee91-5a11-4a4d-8b6d-32273c3c26b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:39.484807 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484736 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.984722634 +0000 UTC m=+1.980423574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:39.487133 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.484680 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ba41133-2ab9-43ac-94d9-adfade07fee6-host\") pod \"tuned-76cvn\" (UID: \"9ba41133-2ab9-43ac-94d9-adfade07fee6\") " pod="openshift-cluster-node-tuning-operator/tuned-76cvn" Nov 02 02:47:39.487133 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484776 4868 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:39.487133 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.484813 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:39.984801649 +0000 UTC m=+1.980502571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:39.520270 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.520251 4868 scope.go:122] "RemoveContainer" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" Nov 02 02:47:39.987964 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.987934 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.988129 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.987972 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:39.988129 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.987998 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.988129 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988109 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:39.988267 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988134 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:39.988267 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988191 4868 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:39.988267 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988202 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988181123 +0000 UTC m=+2.983882071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:39.988267 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988256 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988243281 +0000 UTC m=+2.983944206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:39.988267 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988261 4868 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988305 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988292138 +0000 UTC m=+2.983993059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988324 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988367 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988393 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988410 4868 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988420 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988438 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988428216 +0000 UTC m=+2.984129136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988469 4868 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:39.988490 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988494 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988497 4868 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988500 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988490407 +0000 UTC m=+2.984191327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988535 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988528137 +0000 UTC m=+2.984229057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988553 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988568 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988595 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988589815 +0000 UTC m=+2.984290737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988609 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.98860182 +0000 UTC m=+2.984302740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988556 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988638 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988658 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988714 4868 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988723 4868 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988735 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988728247 +0000 UTC m=+2.984429168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : secret "metrics-daemon-secret" not found Nov 02 02:47:39.988852 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:39.988759 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:40.988746628 +0000 UTC m=+2.984447569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:39.989229 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:39.988875 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d60a499f-81af-466f-9be9-a581feee8523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:40.046587 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.046566 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:47:40.163578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.163534 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:47:40.163959 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.163944 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/1.log" Nov 02 02:47:40.163998 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.163974 4868 generic.go:356] "Generic (PLEG): container finished" podID="971a0263-ee9a-4280-aa61-d385483ca8f0" containerID="36e0f7a98eaa74b0012bf199c313b9f76b2ba1a101021e7a1b2e1c1ad4fabd8b" exitCode=255 Nov 02 02:47:40.164071 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.164051 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" event={"ID":"971a0263-ee9a-4280-aa61-d385483ca8f0","Type":"ContainerDied","Data":"36e0f7a98eaa74b0012bf199c313b9f76b2ba1a101021e7a1b2e1c1ad4fabd8b"} Nov 02 02:47:40.164134 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.164123 4868 scope.go:122] "RemoveContainer" containerID="a9016f242969e4576ac91fa92d06b94903e586a442beeac95c6369624708522e" Nov 02 02:47:40.164630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.164611 4868 scope.go:122] "RemoveContainer" containerID="36e0f7a98eaa74b0012bf199c313b9f76b2ba1a101021e7a1b2e1c1ad4fabd8b" Nov 02 02:47:40.164766 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.164748 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-k6f8b_openshift-console-operator(971a0263-ee9a-4280-aa61-d385483ca8f0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" Nov 02 02:47:40.997735 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997702 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:40.997735 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997738 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997762 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997806 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997831 4868 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997842 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997850 4868 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997886 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.997872753 +0000 UTC m=+4.993573678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997903 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997922 4868 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997932 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997943 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997922 4868 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.997954 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997962 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.997947513 +0000 UTC m=+4.993648439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997962 4868 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.997997 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.997979164 +0000 UTC m=+4.993680096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998009 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998018 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998011765 +0000 UTC m=+4.993712686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998019 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998030 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998024493 +0000 UTC m=+4.993725414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:40.998037 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998048 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998040239 +0000 UTC m=+4.993741167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998060 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998055487 +0000 UTC m=+4.993756407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.998085 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.998110 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:40.998139 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998225 4868 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998258 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998251089 +0000 UTC m=+4.993952011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : secret "metrics-daemon-secret" not found Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998225 4868 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998271 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998263513 +0000 UTC m=+4.993964434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:40.998691 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:40.998283 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:42.998278098 +0000 UTC m=+4.993979018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:41.170131 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:41.170105 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:47:43.017651 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017599 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017669 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017700 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017723 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017748 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017775 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017807 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017816 4868 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017841 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017850 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017874 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017888 4868 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017911 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.017884485 +0000 UTC m=+9.013585442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017941 4868 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017955 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017960 4868 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017967 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.017872 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017917 4868 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017891 4868 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.017945 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.017933569 +0000 UTC m=+9.013634491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018098 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.018076655 +0000 UTC m=+9.013777583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018113 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.018106288 +0000 UTC m=+9.013807210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018143 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.018126024 +0000 UTC m=+9.013826973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018168 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.018158662 +0000 UTC m=+9.013859611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:43.018180 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018199 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.01819182 +0000 UTC m=+9.013892744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:43.019095 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018216 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.018208126 +0000 UTC m=+9.013909116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:43.019095 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018234 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.01822727 +0000 UTC m=+9.013928191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : secret "metrics-daemon-secret" not found Nov 02 02:47:43.019095 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.018264 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:43.019095 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018463 4868 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 02:47:43.019095 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.018525 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert podName:210af9a6-abe4-4617-bf11-5ca72b7da8f3 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:47.018508614 +0000 UTC m=+9.014209559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-dn6x6" (UID: "210af9a6-abe4-4617-bf11-5ca72b7da8f3") : secret "networking-console-plugin-cert" not found Nov 02 02:47:43.975667 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.975635 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:43.976125 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:43.976096 4868 scope.go:122] "RemoveContainer" containerID="36e0f7a98eaa74b0012bf199c313b9f76b2ba1a101021e7a1b2e1c1ad4fabd8b" Nov 02 02:47:43.976321 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:43.976295 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-k6f8b_openshift-console-operator(971a0263-ee9a-4280-aa61-d385483ca8f0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" Nov 02 02:47:44.111626 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:44.111606 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:44.705008 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:44.704973 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfzxg_f7bf0a3b-df4f-458d-b0ae-72483d95f2d7/dns-node-resolver/0.log" Nov 02 02:47:45.709010 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:45.708974 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5sb5c_91547ba7-4770-461b-a34b-0f3a46bdde43/node-ca/0.log" Nov 02 02:47:46.705850 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:46.705815 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rgk6v_50d03c81-d243-4970-9334-023b1af99f91/serve-healthcheck-canary/0.log" Nov 02 02:47:46.955773 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:46.955744 4868 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:46.956176 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:46.956072 4868 scope.go:122] "RemoveContainer" containerID="36e0f7a98eaa74b0012bf199c313b9f76b2ba1a101021e7a1b2e1c1ad4fabd8b" Nov 02 02:47:46.956250 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:46.956222 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-k6f8b_openshift-console-operator(971a0263-ee9a-4280-aa61-d385483ca8f0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" Nov 02 02:47:47.053410 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.053383 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:47.053537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.053416 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:47.053537 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.053461 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:47.053537 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053528 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053547 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-678859df4-mxqvr: secret "image-registry-tls" not found Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053555 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.053536432 +0000 UTC m=+17.049237367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : configmap references non-existent config key: service-ca.crt Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053576 4868 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053614 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls podName:4c9cf6d2-1802-43d0-99ef-2480983f9f52 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.053601091 +0000 UTC m=+17.049302031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls") pod "dns-default-rvk96" (UID: "4c9cf6d2-1802-43d0-99ef-2480983f9f52") : secret "dns-default-metrics-tls" not found Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.053617 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053656 4868 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.053671 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:47.053695 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053679 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs podName:b855c6cb-d565-4687-bd13-248d85111459 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.05367311 +0000 UTC m=+17.049374079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs") pod "network-metrics-daemon-cvzpz" (UID: "b855c6cb-d565-4687-bd13-248d85111459") : secret "metrics-daemon-secret" not found Nov 02 02:47:47.054178 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.053724 4868 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 02:47:47.054270 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054247 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls podName:9d57bc0b-e4b2-401a-a41f-49438f0edb58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.053700928 +0000 UTC m=+17.049401854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls") pod "image-registry-678859df4-mxqvr" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58") : secret "image-registry-tls" not found Nov 02 02:47:47.054411 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054289 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls podName:d60a499f-81af-466f-9be9-a581feee8523 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.054278431 +0000 UTC m=+17.049979363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jsrl" (UID: "d60a499f-81af-466f-9be9-a581feee8523") : secret "insights-runtime-extractor-tls" not found Nov 02 02:47:47.054411 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.054322 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:47.054411 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.054376 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.054409 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.054437 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054447 4868 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054461 4868 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cb8549547-trw7f: secret "image-registry-tls" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.054481 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054502 4868 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054510 4868 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054501 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls podName:5b14c538-49b9-4e01-9bb0-8570712848c0 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.05448883 +0000 UTC m=+17.050189754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls") pod "image-registry-7cb8549547-trw7f" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0") : secret "image-registry-tls" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054545 4868 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054559 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls podName:201c03d0-ed5b-4ccb-b4c2-534fe47511c4 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.054547541 +0000 UTC m=+17.050248466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sgsh7" (UID: "201c03d0-ed5b-4ccb-b4c2-534fe47511c4") : secret "cluster-monitoring-operator-tls" not found Nov 02 02:47:47.054578 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054573 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs podName:d25eaef0-b0ac-4326-925b-aaef13840661 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.054567104 +0000 UTC m=+17.050268024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs") pod "router-default-648db889-4lftz" (UID: "d25eaef0-b0ac-4326-925b-aaef13840661") : secret "router-metrics-certs-default" not found Nov 02 02:47:47.054916 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:47:47.054590 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls podName:4ee35072-3ed9-4c17-9d67-714aec085b58 nodeName:}" failed. No retries permitted until 2026-11-02 02:47:55.05458124 +0000 UTC m=+17.050282160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kcxkq" (UID: "4ee35072-3ed9-4c17-9d67-714aec085b58") : secret "samples-operator-tls" not found Nov 02 02:47:47.054916 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.054788 4868 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 02:47:47.056898 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.056881 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/210af9a6-abe4-4617-bf11-5ca72b7da8f3-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-dn6x6\" (UID: \"210af9a6-abe4-4617-bf11-5ca72b7da8f3\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:47.322693 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.322617 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" Nov 02 02:47:47.507365 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.507332 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2_ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0/kube-storage-version-migrator-operator/0.log" Nov 02 02:47:47.683435 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.683404 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-dn6x6"] Nov 02 02:47:47.687077 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:47.687040 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210af9a6_abe4_4617_bf11_5ca72b7da8f3.slice/crio-7fe228d8af24ce4d99298ed10b0866f9a17b1e3ded5aa6dde67fc77a4d349cfd WatchSource:0}: Error finding container 7fe228d8af24ce4d99298ed10b0866f9a17b1e3ded5aa6dde67fc77a4d349cfd: Status 404 returned error can't find the container with id 7fe228d8af24ce4d99298ed10b0866f9a17b1e3ded5aa6dde67fc77a4d349cfd Nov 02 02:47:47.689315 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:47.689298 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 02:47:48.121181 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:48.121156 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:48.121571 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:48.121334 4868 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 02:47:48.195776 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:48.195744 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" event={"ID":"210af9a6-abe4-4617-bf11-5ca72b7da8f3","Type":"ContainerStarted","Data":"7fe228d8af24ce4d99298ed10b0866f9a17b1e3ded5aa6dde67fc77a4d349cfd"} Nov 02 02:47:48.938327 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:48.938293 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q6wpj" Nov 02 02:47:50.107688 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.107657 4868 scope.go:122] "RemoveContainer" containerID="36e0f7a98eaa74b0012bf199c313b9f76b2ba1a101021e7a1b2e1c1ad4fabd8b" Nov 02 02:47:50.204369 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.204332 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:47:50.204493 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.204438 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" event={"ID":"971a0263-ee9a-4280-aa61-d385483ca8f0","Type":"ContainerStarted","Data":"3c9ad64f3274e17b03de0a91f6ed708f1afa11ae9a63b9bad7e204fde3322c75"} Nov 02 02:47:50.204738 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.204714 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:50.205746 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.205720 4868 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-k6f8b container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body= Nov 02 02:47:50.205847 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.205761 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" podUID="971a0263-ee9a-4280-aa61-d385483ca8f0" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" Nov 02 02:47:50.206025 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:50.206006 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-dn6x6" event={"ID":"210af9a6-abe4-4617-bf11-5ca72b7da8f3","Type":"ContainerStarted","Data":"d9786f70b8d5df00185d9c00049555655f0862be3e9b888e2f23e266edf3c4ff"} Nov 02 02:47:51.213018 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:51.212991 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-k6f8b" Nov 02 02:47:52.439417 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:52.439391 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v8xt2" Nov 02 02:47:55.127843 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.127808 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.127901 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.127941 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.127976 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.128020 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.128065 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.128209 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:55.128274 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.128273 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:55.128757 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.128310 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:55.129398 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.129369 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d25eaef0-b0ac-4326-925b-aaef13840661-service-ca-bundle\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:55.131174 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131135 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d60a499f-81af-466f-9be9-a581feee8523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jsrl\" (UID: \"d60a499f-81af-466f-9be9-a581feee8523\") " pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:55.131282 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131142 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b855c6cb-d565-4687-bd13-248d85111459-metrics-certs\") pod \"network-metrics-daemon-cvzpz\" (UID: \"b855c6cb-d565-4687-bd13-248d85111459\") " pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:55.131330 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131280 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"image-registry-678859df4-mxqvr\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:55.131330 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131283 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"image-registry-7cb8549547-trw7f\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:55.131440 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131280 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25eaef0-b0ac-4326-925b-aaef13840661-metrics-certs\") pod \"router-default-648db889-4lftz\" (UID: \"d25eaef0-b0ac-4326-925b-aaef13840661\") " pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:55.131775 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131751 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/201c03d0-ed5b-4ccb-b4c2-534fe47511c4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sgsh7\" (UID: \"201c03d0-ed5b-4ccb-b4c2-534fe47511c4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:55.131931 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.131912 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c9cf6d2-1802-43d0-99ef-2480983f9f52-metrics-tls\") pod \"dns-default-rvk96\" (UID: \"4c9cf6d2-1802-43d0-99ef-2480983f9f52\") " pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:55.132194 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.132175 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee35072-3ed9-4c17-9d67-714aec085b58-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kcxkq\" (UID: \"4ee35072-3ed9-4c17-9d67-714aec085b58\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:55.132947 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.132930 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jsrl" Nov 02 02:47:55.418869 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.418790 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" Nov 02 02:47:55.419017 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.418879 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:55.419017 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.418930 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" Nov 02 02:47:55.420082 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.420062 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:55.431469 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.431445 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:55.431571 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.431487 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvzpz" Nov 02 02:47:55.432574 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.432532 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:55.512613 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.511983 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jsrl"] Nov 02 02:47:55.539091 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:55.539045 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60a499f_81af_466f_9be9_a581feee8523.slice/crio-88f293a4c92297399ab62e68a926b445b379a92f91effd15cfc810eaf77604ce WatchSource:0}: Error finding container 88f293a4c92297399ab62e68a926b445b379a92f91effd15cfc810eaf77604ce: Status 404 returned error can't find the container with id 88f293a4c92297399ab62e68a926b445b379a92f91effd15cfc810eaf77604ce Nov 02 02:47:55.594643 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.594621 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678859df4-mxqvr"] Nov 02 02:47:55.827637 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.827568 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-648db889-4lftz"] Nov 02 02:47:55.843836 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:55.843813 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25eaef0_b0ac_4326_925b_aaef13840661.slice/crio-6ee9204edf002f5f80e8db8e36364af945f8ccf84e71f6c7a65aadfa11ed7e6c WatchSource:0}: Error finding container 6ee9204edf002f5f80e8db8e36364af945f8ccf84e71f6c7a65aadfa11ed7e6c: Status 404 returned error can't find the container with id 6ee9204edf002f5f80e8db8e36364af945f8ccf84e71f6c7a65aadfa11ed7e6c Nov 02 02:47:55.850651 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.850630 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7"] Nov 02 02:47:55.853597 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:55.853574 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201c03d0_ed5b_4ccb_b4c2_534fe47511c4.slice/crio-d41ea267f549404cc79ad3290e24a22dad8fcd21c157d7f704e988312e241c99 WatchSource:0}: Error finding container d41ea267f549404cc79ad3290e24a22dad8fcd21c157d7f704e988312e241c99: Status 404 returned error can't find the container with id d41ea267f549404cc79ad3290e24a22dad8fcd21c157d7f704e988312e241c99 Nov 02 02:47:55.878006 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.877982 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cvzpz"] Nov 02 02:47:55.879607 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.879589 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq"] Nov 02 02:47:55.879654 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.879644 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvk96"] Nov 02 02:47:55.881209 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:55.881147 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9cf6d2_1802_43d0_99ef_2480983f9f52.slice/crio-e1b827cacbf18b9341c537062862e237bd3827e10b54e625fb53cc0be81fe1ad WatchSource:0}: Error finding container e1b827cacbf18b9341c537062862e237bd3827e10b54e625fb53cc0be81fe1ad: Status 404 returned error can't find the container with id e1b827cacbf18b9341c537062862e237bd3827e10b54e625fb53cc0be81fe1ad Nov 02 02:47:55.908055 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.908032 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cb8549547-trw7f"] Nov 02 02:47:55.908783 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.908761 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678859df4-mxqvr"] Nov 02 02:47:55.914542 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:47:55.914516 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b14c538_49b9_4e01_9bb0_8570712848c0.slice/crio-a35dd7139ef0fe67392afe69fbeb192ebf42e3698140fd6e046cd1e72375ea19 WatchSource:0}: Error finding container a35dd7139ef0fe67392afe69fbeb192ebf42e3698140fd6e046cd1e72375ea19: Status 404 returned error can't find the container with id a35dd7139ef0fe67392afe69fbeb192ebf42e3698140fd6e046cd1e72375ea19 Nov 02 02:47:55.916182 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:55.916167 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cb8549547-trw7f"] Nov 02 02:47:56.226745 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.226700 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-648db889-4lftz" event={"ID":"d25eaef0-b0ac-4326-925b-aaef13840661","Type":"ContainerStarted","Data":"eddb7825a77f7c8314e9013d8774ba4def45be7c17a8625136067ec925a0518d"} Nov 02 02:47:56.227202 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.226763 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-648db889-4lftz" event={"ID":"d25eaef0-b0ac-4326-925b-aaef13840661","Type":"ContainerStarted","Data":"6ee9204edf002f5f80e8db8e36364af945f8ccf84e71f6c7a65aadfa11ed7e6c"} Nov 02 02:47:56.229340 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.229230 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jsrl" event={"ID":"d60a499f-81af-466f-9be9-a581feee8523","Type":"ContainerStarted","Data":"e74bff5532a9d983664bcee2c7e6083f8b7a0bb33c83af6e5bf1dcad563e8a91"} Nov 02 02:47:56.229340 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.229290 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jsrl" event={"ID":"d60a499f-81af-466f-9be9-a581feee8523","Type":"ContainerStarted","Data":"88f293a4c92297399ab62e68a926b445b379a92f91effd15cfc810eaf77604ce"} Nov 02 02:47:56.231935 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.231904 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" event={"ID":"5b14c538-49b9-4e01-9bb0-8570712848c0","Type":"ContainerStarted","Data":"79a05291608432a62fb8d2f9725a2d1827b2f415140e1ece420fda1fbd2cd742"} Nov 02 02:47:56.232058 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.231940 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" event={"ID":"5b14c538-49b9-4e01-9bb0-8570712848c0","Type":"ContainerStarted","Data":"a35dd7139ef0fe67392afe69fbeb192ebf42e3698140fd6e046cd1e72375ea19"} Nov 02 02:47:56.232058 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.231989 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:47:56.233631 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.233603 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" event={"ID":"4ee35072-3ed9-4c17-9d67-714aec085b58","Type":"ContainerStarted","Data":"adbd260615a518bc6a97153012c66f1568fd83d84a232a3b14bba6d80583eb67"} Nov 02 02:47:56.235168 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.235140 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvk96" event={"ID":"4c9cf6d2-1802-43d0-99ef-2480983f9f52","Type":"ContainerStarted","Data":"e1b827cacbf18b9341c537062862e237bd3827e10b54e625fb53cc0be81fe1ad"} Nov 02 02:47:56.237147 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.237117 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678859df4-mxqvr" event={"ID":"9d57bc0b-e4b2-401a-a41f-49438f0edb58","Type":"ContainerStarted","Data":"69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad"} Nov 02 02:47:56.237251 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.237160 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678859df4-mxqvr" event={"ID":"9d57bc0b-e4b2-401a-a41f-49438f0edb58","Type":"ContainerStarted","Data":"4c9a3bd48948915f914b88222de4aaaa25c55bc9c91e344215cc26bb9aed8300"} Nov 02 02:47:56.237251 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.237229 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:47:56.238750 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.238523 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cvzpz" event={"ID":"b855c6cb-d565-4687-bd13-248d85111459","Type":"ContainerStarted","Data":"45e112e2eb5b1211240d1072e7f4ee9b1007e83ac7df49194ad5ad69dbcd3943"} Nov 02 02:47:56.239897 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.239873 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" event={"ID":"201c03d0-ed5b-4ccb-b4c2-534fe47511c4","Type":"ContainerStarted","Data":"d41ea267f549404cc79ad3290e24a22dad8fcd21c157d7f704e988312e241c99"} Nov 02 02:47:56.420525 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.420496 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:56.422469 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:56.422445 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:57.250938 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:57.250898 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jsrl" event={"ID":"d60a499f-81af-466f-9be9-a581feee8523","Type":"ContainerStarted","Data":"b2d2d2daf527769a5fb59578f0a76224e6f346461d62502f2ec5325e59d7355f"} Nov 02 02:47:57.251423 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:57.250978 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:57.252581 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:57.252515 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-648db889-4lftz" Nov 02 02:47:58.271315 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:58.271269 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cvzpz" event={"ID":"b855c6cb-d565-4687-bd13-248d85111459","Type":"ContainerStarted","Data":"77caaf58bf44909509a67ad939184c3ea8f754f7f191929fee5c2098fbe03735"} Nov 02 02:47:58.271772 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:58.271492 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cvzpz" event={"ID":"b855c6cb-d565-4687-bd13-248d85111459","Type":"ContainerStarted","Data":"c51416d41cc3ed457d957bda5b7af42f810e60759be8b665506156d6ed26392e"} Nov 02 02:47:59.278238 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.278196 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvk96" event={"ID":"4c9cf6d2-1802-43d0-99ef-2480983f9f52","Type":"ContainerStarted","Data":"6d574d08a455379e8a39bd8f0bf43d50d2b3206305718714fb4d4bfd73d4563e"} Nov 02 02:47:59.278681 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.278265 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvk96" event={"ID":"4c9cf6d2-1802-43d0-99ef-2480983f9f52","Type":"ContainerStarted","Data":"cd161363458bd545e27ea92b5c0e6c7dba431e746c69e2560b19a30784c58fec"} Nov 02 02:47:59.278980 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.278960 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rvk96" Nov 02 02:47:59.280607 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.280580 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sgsh7" event={"ID":"201c03d0-ed5b-4ccb-b4c2-534fe47511c4","Type":"ContainerStarted","Data":"38d57b0ebc8840ca472175e3a3f25c4a77d1a25288697c870cfac2053e73275c"} Nov 02 02:47:59.282733 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.282702 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jsrl" event={"ID":"d60a499f-81af-466f-9be9-a581feee8523","Type":"ContainerStarted","Data":"1dd7af85a21ea4ae6e2e52ac51adff879b3e6fb07834709b04bbbff194a33d57"} Nov 02 02:47:59.284395 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.284347 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" event={"ID":"4ee35072-3ed9-4c17-9d67-714aec085b58","Type":"ContainerStarted","Data":"ae4820fdf5364c9f257d80a8f62bd5770c071bdced136a4ca9cb7634cad4b15b"} Nov 02 02:47:59.284526 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:47:59.284412 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kcxkq" event={"ID":"4ee35072-3ed9-4c17-9d67-714aec085b58","Type":"ContainerStarted","Data":"b58678a698932d5d93ed1600ea1a1b847fa554ad61490ffb440c9d958afd8924"} Nov 02 02:48:03.446721 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:48:03.446675 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:48:10.167067 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:10.167037 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d42js" Nov 02 02:48:10.293773 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:10.293741 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rvk96" Nov 02 02:48:11.394339 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.394300 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2gs9s"] Nov 02 02:48:11.445012 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.444987 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.553143 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.553119 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 02:48:11.553294 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.553254 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gtx4f\"" Nov 02 02:48:11.553378 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.553314 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 02:48:11.558616 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558598 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558684 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558626 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldf85\" (UniqueName: \"kubernetes.io/projected/fee48a3f-91ff-420f-bf42-8acc0d7877a9-kube-api-access-ldf85\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558684 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558647 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-textfile\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558684 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558662 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-tls\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558824 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558750 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee48a3f-91ff-420f-bf42-8acc0d7877a9-metrics-client-ca\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558824 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558813 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-root\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558854 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-wtmp\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558893 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558873 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-sys\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.558965 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.558901 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.602905 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.602874 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 02:48:11.603081 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.603005 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 02:48:11.659288 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659226 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-wtmp\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659288 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659267 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-sys\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659290 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659334 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659376 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-sys\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659382 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldf85\" (UniqueName: \"kubernetes.io/projected/fee48a3f-91ff-420f-bf42-8acc0d7877a9-kube-api-access-ldf85\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659426 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-textfile\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659438 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-wtmp\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659449 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-tls\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659482 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee48a3f-91ff-420f-bf42-8acc0d7877a9-metrics-client-ca\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659910 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659537 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-root\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.659910 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659621 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fee48a3f-91ff-420f-bf42-8acc0d7877a9-root\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.660020 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.659939 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-textfile\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.660078 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.660047 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.660162 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.660143 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fee48a3f-91ff-420f-bf42-8acc0d7877a9-metrics-client-ca\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.661797 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.661781 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-tls\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.661989 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.661969 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fee48a3f-91ff-420f-bf42-8acc0d7877a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.715505 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.715479 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldf85\" (UniqueName: \"kubernetes.io/projected/fee48a3f-91ff-420f-bf42-8acc0d7877a9-kube-api-access-ldf85\") pod \"node-exporter-2gs9s\" (UID: \"fee48a3f-91ff-420f-bf42-8acc0d7877a9\") " pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.796301 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:11.796278 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2gs9s" Nov 02 02:48:11.836803 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:48:11.836780 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee48a3f_91ff_420f_bf42_8acc0d7877a9.slice/crio-a56fa6bcb6b1eac39851043c97b669b78d904d1acf4387023b4f4a7532426ae9 WatchSource:0}: Error finding container a56fa6bcb6b1eac39851043c97b669b78d904d1acf4387023b4f4a7532426ae9: Status 404 returned error can't find the container with id a56fa6bcb6b1eac39851043c97b669b78d904d1acf4387023b4f4a7532426ae9 Nov 02 02:48:12.328188 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:12.328147 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2gs9s" event={"ID":"fee48a3f-91ff-420f-bf42-8acc0d7877a9","Type":"ContainerStarted","Data":"a56fa6bcb6b1eac39851043c97b669b78d904d1acf4387023b4f4a7532426ae9"} Nov 02 02:48:14.338080 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:14.338003 4868 generic.go:356] "Generic (PLEG): container finished" podID="fee48a3f-91ff-420f-bf42-8acc0d7877a9" containerID="7c7d631c26af375fc411d3c0f7f4c47ff775037a48b47fbc47c105a4686a9373" exitCode=0 Nov 02 02:48:14.338439 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:14.338087 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2gs9s" event={"ID":"fee48a3f-91ff-420f-bf42-8acc0d7877a9","Type":"ContainerDied","Data":"7c7d631c26af375fc411d3c0f7f4c47ff775037a48b47fbc47c105a4686a9373"} Nov 02 02:48:15.343872 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:15.343833 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2gs9s" event={"ID":"fee48a3f-91ff-420f-bf42-8acc0d7877a9","Type":"ContainerStarted","Data":"a325f5a0b03a6acedbf2d5ec64155be6298fce52ea936d563b6954de25666882"} Nov 02 02:48:15.344394 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:15.343899 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2gs9s" event={"ID":"fee48a3f-91ff-420f-bf42-8acc0d7877a9","Type":"ContainerStarted","Data":"0b8364a745a0c17c62bf05884732010cf2dc0c300ff5da2142437aa9e4c92794"} Nov 02 02:48:15.512861 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:15.512797 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2gs9s" podStartSLOduration=2.391566816 podStartE2EDuration="4.512779317s" totalImagesPullingTime="2.040568335s" totalInitContainerRuntime="80.644166ms" isStatefulPod=true podCreationTimestamp="2026-11-02 02:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:48:15.51162271 +0000 UTC m=+37.507323675" watchObservedRunningTime="2026-11-02 02:48:15.512779317 +0000 UTC m=+37.508480259" Nov 02 02:48:16.237099 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:16.237071 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:48:16.241900 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:16.241875 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:48:21.252778 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.252675 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" podUID="5b14c538-49b9-4e01-9bb0-8570712848c0" containerName="registry" containerID="cri-o://79a05291608432a62fb8d2f9725a2d1827b2f415140e1ece420fda1fbd2cd742" gracePeriod=30 Nov 02 02:48:21.256188 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.256158 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-678859df4-mxqvr" podUID="9d57bc0b-e4b2-401a-a41f-49438f0edb58" containerName="registry" containerID="cri-o://69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad" gracePeriod=30 Nov 02 02:48:21.368863 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.368834 4868 generic.go:356] "Generic (PLEG): container finished" podID="5b14c538-49b9-4e01-9bb0-8570712848c0" containerID="79a05291608432a62fb8d2f9725a2d1827b2f415140e1ece420fda1fbd2cd742" exitCode=0 Nov 02 02:48:21.368969 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.368865 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" event={"ID":"5b14c538-49b9-4e01-9bb0-8570712848c0","Type":"ContainerDied","Data":"79a05291608432a62fb8d2f9725a2d1827b2f415140e1ece420fda1fbd2cd742"} Nov 02 02:48:21.497243 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.497218 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:48:21.499932 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.499914 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:48:21.633740 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.633712 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.633868 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.633750 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.633868 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.633769 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.633868 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.633807 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.634030 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.633965 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.634030 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634020 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.634133 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634064 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.634133 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634123 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.634236 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634160 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634236 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634191 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634236 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634227 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634486 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634270 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634486 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634291 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:48:21.634486 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634300 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:48:21.634486 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634306 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") pod \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\" (UID: \"9d57bc0b-e4b2-401a-a41f-49438f0edb58\") " Nov 02 02:48:21.634486 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634420 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634486 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634473 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634871 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634650 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") pod \"5b14c538-49b9-4e01-9bb0-8570712848c0\" (UID: \"5b14c538-49b9-4e01-9bb0-8570712848c0\") " Nov 02 02:48:21.634871 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634851 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:48:21.634976 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634933 4868 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-trusted-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.634976 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634956 4868 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-certificates\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.635373 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.634973 4868 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-trusted-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.636099 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.636073 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:48:21.637006 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.636976 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:48:21.637257 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.637233 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:48:21.637410 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.637386 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "kube-api-access-8452z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:48:21.638109 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.638082 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:48:21.638201 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.638120 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "kube-api-access-mmthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:48:21.638492 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.638467 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:48:21.638850 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.638820 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:48:21.639077 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.638952 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:48:21.639524 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.639276 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:48:21.639648 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.639626 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:48:21.643467 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.643443 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted" pod "9d57bc0b-e4b2-401a-a41f-49438f0edb58" (UID: "9d57bc0b-e4b2-401a-a41f-49438f0edb58"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:48:21.644056 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.644037 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted" pod "5b14c538-49b9-4e01-9bb0-8570712848c0" (UID: "5b14c538-49b9-4e01-9bb0-8570712848c0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:48:21.736720 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736689 4868 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-bound-sa-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736720 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736716 4868 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-installation-pull-secrets\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736732 4868 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-bound-sa-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736745 4868 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736759 4868 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-image-registry-private-configuration\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736773 4868 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b14c538-49b9-4e01-9bb0-8570712848c0-ca-trust-extracted\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736787 4868 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b14c538-49b9-4e01-9bb0-8570712848c0-registry-certificates\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736799 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8452z\" (UniqueName: \"kubernetes.io/projected/5b14c538-49b9-4e01-9bb0-8570712848c0-kube-api-access-8452z\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736814 4868 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d57bc0b-e4b2-401a-a41f-49438f0edb58-ca-trust-extracted\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736827 4868 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b14c538-49b9-4e01-9bb0-8570712848c0-image-registry-private-configuration\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736841 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmthm\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-kube-api-access-mmthm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736855 4868 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d57bc0b-e4b2-401a-a41f-49438f0edb58-registry-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:21.736907 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:21.736869 4868 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d57bc0b-e4b2-401a-a41f-49438f0edb58-installation-pull-secrets\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:48:22.374155 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.374088 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" Nov 02 02:48:22.374155 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.374139 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cb8549547-trw7f" event={"ID":"5b14c538-49b9-4e01-9bb0-8570712848c0","Type":"ContainerDied","Data":"a35dd7139ef0fe67392afe69fbeb192ebf42e3698140fd6e046cd1e72375ea19"} Nov 02 02:48:22.374617 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.374209 4868 scope.go:122] "RemoveContainer" containerID="79a05291608432a62fb8d2f9725a2d1827b2f415140e1ece420fda1fbd2cd742" Nov 02 02:48:22.375630 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.375609 4868 generic.go:356] "Generic (PLEG): container finished" podID="9d57bc0b-e4b2-401a-a41f-49438f0edb58" containerID="69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad" exitCode=0 Nov 02 02:48:22.375741 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.375643 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678859df4-mxqvr" event={"ID":"9d57bc0b-e4b2-401a-a41f-49438f0edb58","Type":"ContainerDied","Data":"69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad"} Nov 02 02:48:22.375741 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.375663 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678859df4-mxqvr" Nov 02 02:48:22.375834 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.375668 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678859df4-mxqvr" event={"ID":"9d57bc0b-e4b2-401a-a41f-49438f0edb58","Type":"ContainerDied","Data":"4c9a3bd48948915f914b88222de4aaaa25c55bc9c91e344215cc26bb9aed8300"} Nov 02 02:48:22.382248 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.382232 4868 scope.go:122] "RemoveContainer" containerID="69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad" Nov 02 02:48:22.389028 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.389010 4868 scope.go:122] "RemoveContainer" containerID="69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad" Nov 02 02:48:22.389283 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:48:22.389264 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad\": container with ID starting with 69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad not found: ID does not exist" containerID="69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad" Nov 02 02:48:22.389329 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.389296 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad"} err="failed to get container status \"69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad\": rpc error: code = NotFound desc = could not find container \"69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad\": container with ID starting with 69bc16099132838812054c395fb56c0bebdd78de9191b75668e46346a7db97ad not found: ID does not exist" Nov 02 02:48:22.585809 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.585768 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-678859df4-mxqvr"] Nov 02 02:48:22.634890 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.634836 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-678859df4-mxqvr"] Nov 02 02:48:22.697323 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.697300 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cb8549547-trw7f"] Nov 02 02:48:22.702261 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:22.702240 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cb8549547-trw7f"] Nov 02 02:48:24.112514 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:24.112478 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b14c538-49b9-4e01-9bb0-8570712848c0" path="/var/lib/kubelet/pods/5b14c538-49b9-4e01-9bb0-8570712848c0/volumes" Nov 02 02:48:24.112974 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:24.112888 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d57bc0b-e4b2-401a-a41f-49438f0edb58" path="/var/lib/kubelet/pods/9d57bc0b-e4b2-401a-a41f-49438f0edb58/volumes" Nov 02 02:48:33.444470 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:48:33.444432 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:48:49.475695 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:49.475665 4868 generic.go:356] "Generic (PLEG): container finished" podID="c94082a0-1953-42aa-9615-7820c50317eb" containerID="d9d39a83d52fb021c0fd068ae34724fe666d36bd64e7fe75c065e16df0666e9d" exitCode=0 Nov 02 02:48:49.476042 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:49.475742 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" event={"ID":"c94082a0-1953-42aa-9615-7820c50317eb","Type":"ContainerDied","Data":"d9d39a83d52fb021c0fd068ae34724fe666d36bd64e7fe75c065e16df0666e9d"} Nov 02 02:48:49.476097 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:49.476069 4868 scope.go:122] "RemoveContainer" containerID="d9d39a83d52fb021c0fd068ae34724fe666d36bd64e7fe75c065e16df0666e9d" Nov 02 02:48:49.477317 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:49.477292 4868 generic.go:356] "Generic (PLEG): container finished" podID="ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0" containerID="c9a24da030f3ea94237038f114aec7de0f48939399f6a4a2b0bc425a16f2ec44" exitCode=0 Nov 02 02:48:49.477436 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:49.477376 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" event={"ID":"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0","Type":"ContainerDied","Data":"c9a24da030f3ea94237038f114aec7de0f48939399f6a4a2b0bc425a16f2ec44"} Nov 02 02:48:49.477678 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:49.477662 4868 scope.go:122] "RemoveContainer" containerID="c9a24da030f3ea94237038f114aec7de0f48939399f6a4a2b0bc425a16f2ec44" Nov 02 02:48:50.484092 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:50.484057 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vhbq" event={"ID":"c94082a0-1953-42aa-9615-7820c50317eb","Type":"ContainerStarted","Data":"85728f115dafd0a98356a5a527e08beff9bb4516862a17c09447840b711b37fd"} Nov 02 02:48:50.485811 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:50.485786 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-kf4t2" event={"ID":"ba6abdd8-3b6c-40a6-8cd5-20229ff88bb0","Type":"ContainerStarted","Data":"9fd6bb4d9c0e920ed98def305aeb31edb7f1e9b3807c1957f8c3077f38ca9010"} Nov 02 02:48:54.501206 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:54.501116 4868 generic.go:356] "Generic (PLEG): container finished" podID="17b96028-853b-4750-a2b3-09658c06aa2e" containerID="b2f5b5002254c35420fea1621da20eb1b8220e69200838f096395d8e187dd0ca" exitCode=0 Nov 02 02:48:54.501694 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:54.501198 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-djgk5" event={"ID":"17b96028-853b-4750-a2b3-09658c06aa2e","Type":"ContainerDied","Data":"b2f5b5002254c35420fea1621da20eb1b8220e69200838f096395d8e187dd0ca"} Nov 02 02:48:54.501694 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:54.501639 4868 scope.go:122] "RemoveContainer" containerID="b2f5b5002254c35420fea1621da20eb1b8220e69200838f096395d8e187dd0ca" Nov 02 02:48:55.506039 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:48:55.505996 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-djgk5" event={"ID":"17b96028-853b-4750-a2b3-09658c06aa2e","Type":"ContainerStarted","Data":"519a81d2fe87db58f7486be86b14d2e495738067543a1803c9e6df3edd460b8b"} Nov 02 02:49:03.445415 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:49:03.445375 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:49:33.444642 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:49:33.444602 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:50:03.445343 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:50:03.445251 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:50:33.444507 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:50:33.444461 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:51:03.446803 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:51:03.446767 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:51:18.051288 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051255 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8"] Nov 02 02:51:18.051807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051583 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9d57bc0b-e4b2-401a-a41f-49438f0edb58" containerName="registry" Nov 02 02:51:18.051807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051597 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d57bc0b-e4b2-401a-a41f-49438f0edb58" containerName="registry" Nov 02 02:51:18.051807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051612 4868 cpu_manager.go:414] "containerMap: removing container" podUID="5b14c538-49b9-4e01-9bb0-8570712848c0" containerName="registry" Nov 02 02:51:18.051807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051619 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b14c538-49b9-4e01-9bb0-8570712848c0" containerName="registry" Nov 02 02:51:18.051807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051676 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b14c538-49b9-4e01-9bb0-8570712848c0" containerName="registry" Nov 02 02:51:18.051807 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.051688 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d57bc0b-e4b2-401a-a41f-49438f0edb58" containerName="registry" Nov 02 02:51:18.054815 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.054794 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.102427 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.102406 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996"] Nov 02 02:51:18.105474 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.105460 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.111089 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.111060 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.111239 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.111187 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.111239 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.111223 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdrm\" (UniqueName: \"kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.151270 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.151246 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hds2w\"" Nov 02 02:51:18.151270 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.151249 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 02:51:18.151270 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.151249 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 02:51:18.157416 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.157394 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8"] Nov 02 02:51:18.197062 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.197038 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996"] Nov 02 02:51:18.206585 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.206566 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wvj9w\"" Nov 02 02:51:18.211629 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.211609 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.211708 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.211642 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.211708 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.211697 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpll\" (UniqueName: \"kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.211813 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.211793 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.211935 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.211911 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.212171 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.211955 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdrm\" (UniqueName: \"kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.212404 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.212384 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.212479 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.212426 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.271851 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.271830 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdrm\" (UniqueName: \"kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.312978 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.312926 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.312978 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.312972 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpll\" (UniqueName: \"kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.313093 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.313001 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.313283 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.313269 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.313321 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.313278 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.370242 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.370222 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:18.373944 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.373924 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpll\" (UniqueName: \"kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.424222 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.424196 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:18.425788 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.425764 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j"] Nov 02 02:51:18.429287 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.429265 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.483190 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.481793 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 02:51:18.483190 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.481868 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 02:51:18.487364 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.487040 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9w67j\"" Nov 02 02:51:18.490736 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.490715 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j"] Nov 02 02:51:18.513942 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.513914 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hhs\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.514055 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.513948 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.514055 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.513966 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.514157 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.514048 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.514157 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.514125 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.514249 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.514158 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.514249 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.514186 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.581700 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.581643 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 02:51:18.581789 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.581699 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 02:51:18.614830 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614810 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.614919 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614844 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hhs\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.614919 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614863 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.614919 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614880 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.614919 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614894 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.615124 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614922 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.615124 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.614964 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.615573 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.615549 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.617139 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.617123 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.617285 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.617269 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.617624 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.617601 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.617674 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.617637 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.670938 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.670917 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hhs\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.671185 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.671166 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2kx7j\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.735767 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.735748 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8"] Nov 02 02:51:18.738098 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:51:18.738077 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd5eb8b_a684_468d_b277_b07bd931e1e1.slice/crio-ab2772971d19cb6c9a1082aeb176ebb66e7b0f6d1e98c04c3e33b39a635f6f73 WatchSource:0}: Error finding container ab2772971d19cb6c9a1082aeb176ebb66e7b0f6d1e98c04c3e33b39a635f6f73: Status 404 returned error can't find the container with id ab2772971d19cb6c9a1082aeb176ebb66e7b0f6d1e98c04c3e33b39a635f6f73 Nov 02 02:51:18.753305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.753288 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:18.842076 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:18.842052 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996"] Nov 02 02:51:18.844425 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:51:18.844362 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1bb4ff_9e7b_4fbb_8e4a_92e19ab2a07d.slice/crio-a051ebe49ad56911858cd784427b8b9ece5a5ebbe74d6e2aa045ab99e0c38d97 WatchSource:0}: Error finding container a051ebe49ad56911858cd784427b8b9ece5a5ebbe74d6e2aa045ab99e0c38d97: Status 404 returned error can't find the container with id a051ebe49ad56911858cd784427b8b9ece5a5ebbe74d6e2aa045ab99e0c38d97 Nov 02 02:51:19.021881 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:19.021848 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" event={"ID":"0bd5eb8b-a684-468d-b277-b07bd931e1e1","Type":"ContainerStarted","Data":"ab2772971d19cb6c9a1082aeb176ebb66e7b0f6d1e98c04c3e33b39a635f6f73"} Nov 02 02:51:19.022809 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:19.022788 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" event={"ID":"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d","Type":"ContainerStarted","Data":"a051ebe49ad56911858cd784427b8b9ece5a5ebbe74d6e2aa045ab99e0c38d97"} Nov 02 02:51:19.119399 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:19.119378 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j"] Nov 02 02:51:19.121770 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:51:19.121746 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6759b1eb_54b3_4ad2_90eb_6e9430a94b31.slice/crio-42fcdcc7f6f1b1844000ec92164b2a99218f162caa1990155536fc4edca554aa WatchSource:0}: Error finding container 42fcdcc7f6f1b1844000ec92164b2a99218f162caa1990155536fc4edca554aa: Status 404 returned error can't find the container with id 42fcdcc7f6f1b1844000ec92164b2a99218f162caa1990155536fc4edca554aa Nov 02 02:51:20.029214 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:20.029179 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" event={"ID":"6759b1eb-54b3-4ad2-90eb-6e9430a94b31","Type":"ContainerStarted","Data":"42fcdcc7f6f1b1844000ec92164b2a99218f162caa1990155536fc4edca554aa"} Nov 02 02:51:23.307319 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:23.307287 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:51:23.307657 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:23.307347 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:51:24.058372 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:24.058287 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" event={"ID":"6759b1eb-54b3-4ad2-90eb-6e9430a94b31","Type":"ContainerStarted","Data":"1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620"} Nov 02 02:51:24.059304 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:24.059264 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:24.061613 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:24.061329 4868 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2kx7j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:51:24.061613 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:24.061394 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:51:24.264259 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:24.264210 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" podStartSLOduration=2.081283497 podStartE2EDuration="6.264196925s" totalImagesPullingTime="4.182913428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:24.223471503 +0000 UTC m=+226.219172444" watchObservedRunningTime="2026-11-02 02:51:24.264196925 +0000 UTC m=+226.259897867" Nov 02 02:51:25.063692 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:25.063650 4868 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2kx7j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:51:25.064172 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:25.063751 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:51:26.067477 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:26.067445 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:51:27.072762 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:27.072725 4868 generic.go:356] "Generic (PLEG): container finished" podID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerID="9f22543321b5654e62c53ecf85a9caaeba0329a62dc052b625b456d89354f0aa" exitCode=0 Nov 02 02:51:27.073209 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:27.072810 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" event={"ID":"0bd5eb8b-a684-468d-b277-b07bd931e1e1","Type":"ContainerDied","Data":"9f22543321b5654e62c53ecf85a9caaeba0329a62dc052b625b456d89354f0aa"} Nov 02 02:51:28.078410 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:28.078373 4868 generic.go:356] "Generic (PLEG): container finished" podID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerID="dfc19e66648299c3fdf7aeffb64f1b2e0e45300cc291fd882f8936ac47ae95f7" exitCode=0 Nov 02 02:51:28.078869 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:28.078433 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" event={"ID":"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d","Type":"ContainerDied","Data":"dfc19e66648299c3fdf7aeffb64f1b2e0e45300cc291fd882f8936ac47ae95f7"} Nov 02 02:51:30.088866 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:30.088837 4868 generic.go:356] "Generic (PLEG): container finished" podID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerID="07f733fc6dabdb79a38309333dcba901d0ede74d426f0d7082b3daaa902b286f" exitCode=0 Nov 02 02:51:30.089169 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:30.088921 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" event={"ID":"0bd5eb8b-a684-468d-b277-b07bd931e1e1","Type":"ContainerDied","Data":"07f733fc6dabdb79a38309333dcba901d0ede74d426f0d7082b3daaa902b286f"} Nov 02 02:51:32.099635 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:32.099603 4868 generic.go:356] "Generic (PLEG): container finished" podID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerID="2cda9bcaf4ade3df3d215c0fcd6c764f26852d63730963a4519891fcbb57c5ac" exitCode=0 Nov 02 02:51:32.100072 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:32.099669 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" event={"ID":"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d","Type":"ContainerDied","Data":"2cda9bcaf4ade3df3d215c0fcd6c764f26852d63730963a4519891fcbb57c5ac"} Nov 02 02:51:33.452342 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:51:33.452308 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:51:38.132453 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:38.132417 4868 generic.go:356] "Generic (PLEG): container finished" podID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerID="2971e022a63a13de72b0d1e453c8b8587d0afa43e1f58a49acdea40ccba926fe" exitCode=0 Nov 02 02:51:38.132929 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:38.132476 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" event={"ID":"0bd5eb8b-a684-468d-b277-b07bd931e1e1","Type":"ContainerDied","Data":"2971e022a63a13de72b0d1e453c8b8587d0afa43e1f58a49acdea40ccba926fe"} Nov 02 02:51:39.143103 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.143069 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" event={"ID":"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d","Type":"ContainerStarted","Data":"9cf1d039e4b4ff8ca271c2360e167987a12a5fd2a08237ed10b7725ae4c4a36b"} Nov 02 02:51:39.257771 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.257707 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" podStartSLOduration=2.403920416 podStartE2EDuration="21.257687018s" totalImagesPullingTime="18.853766602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:51:39.25676223 +0000 UTC m=+241.252463174" watchObservedRunningTime="2026-11-02 02:51:39.257687018 +0000 UTC m=+241.253387962" Nov 02 02:51:39.282902 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.282880 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:39.400460 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.400429 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle\") pod \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " Nov 02 02:51:39.400598 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.400491 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm\" (UniqueName: \"kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm\") pod \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " Nov 02 02:51:39.400598 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.400513 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util\") pod \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\" (UID: \"0bd5eb8b-a684-468d-b277-b07bd931e1e1\") " Nov 02 02:51:39.400934 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.400906 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle" pod "0bd5eb8b-a684-468d-b277-b07bd931e1e1" (UID: "0bd5eb8b-a684-468d-b277-b07bd931e1e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:51:39.403627 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.403603 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm" pod "0bd5eb8b-a684-468d-b277-b07bd931e1e1" (UID: "0bd5eb8b-a684-468d-b277-b07bd931e1e1"). InnerVolumeSpecName "kube-api-access-csdrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:51:39.408229 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.408194 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util" pod "0bd5eb8b-a684-468d-b277-b07bd931e1e1" (UID: "0bd5eb8b-a684-468d-b277-b07bd931e1e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:51:39.501739 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.501688 4868 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-bundle\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:51:39.501739 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.501713 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdrm\" (UniqueName: \"kubernetes.io/projected/0bd5eb8b-a684-468d-b277-b07bd931e1e1-kube-api-access-csdrm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:51:39.501739 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:39.501724 4868 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0bd5eb8b-a684-468d-b277-b07bd931e1e1-util\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:51:40.148204 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:40.148179 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" Nov 02 02:51:40.148581 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:40.148197 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427cv8" event={"ID":"0bd5eb8b-a684-468d-b277-b07bd931e1e1","Type":"ContainerDied","Data":"ab2772971d19cb6c9a1082aeb176ebb66e7b0f6d1e98c04c3e33b39a635f6f73"} Nov 02 02:51:40.148581 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:40.148239 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab2772971d19cb6c9a1082aeb176ebb66e7b0f6d1e98c04c3e33b39a635f6f73" Nov 02 02:51:40.150578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:40.150555 4868 generic.go:356] "Generic (PLEG): container finished" podID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerID="9cf1d039e4b4ff8ca271c2360e167987a12a5fd2a08237ed10b7725ae4c4a36b" exitCode=0 Nov 02 02:51:40.150679 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:40.150634 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" event={"ID":"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d","Type":"ContainerDied","Data":"9cf1d039e4b4ff8ca271c2360e167987a12a5fd2a08237ed10b7725ae4c4a36b"} Nov 02 02:51:41.347499 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.347470 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:41.416899 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.416846 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll\" (UniqueName: \"kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll\") pod \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " Nov 02 02:51:41.416899 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.416888 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle\") pod \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " Nov 02 02:51:41.417035 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.416928 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util\") pod \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\" (UID: \"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d\") " Nov 02 02:51:41.417525 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.417500 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle" pod "eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" (UID: "eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:51:41.418830 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.418804 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll" pod "eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" (UID: "eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d"). InnerVolumeSpecName "kube-api-access-rdpll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:51:41.421165 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.421146 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util" pod "eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" (UID: "eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:51:41.517941 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.517917 4868 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-bundle\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:51:41.517941 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.517939 4868 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-util\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:51:41.518064 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:41.517949 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpll\" (UniqueName: \"kubernetes.io/projected/eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d-kube-api-access-rdpll\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:51:42.160788 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:42.160763 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" event={"ID":"eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d","Type":"ContainerDied","Data":"a051ebe49ad56911858cd784427b8b9ece5a5ebbe74d6e2aa045ab99e0c38d97"} Nov 02 02:51:42.160788 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:42.160780 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfn996" Nov 02 02:51:42.160940 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:42.160802 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a051ebe49ad56911858cd784427b8b9ece5a5ebbe74d6e2aa045ab99e0c38d97" Nov 02 02:51:59.115120 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115088 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b"] Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115379 4868 cpu_manager.go:414] "containerMap: removing container" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="extract" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115392 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="extract" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115403 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="extract" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115408 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="extract" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115415 4868 cpu_manager.go:414] "containerMap: removing container" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="util" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115420 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="util" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115437 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="pull" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115441 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="pull" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115450 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="util" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115455 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="util" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115466 4868 cpu_manager.go:414] "containerMap: removing container" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="pull" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115471 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="pull" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115513 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1bb4ff-9e7b-4fbb-8e4a-92e19ab2a07d" containerName="extract" Nov 02 02:51:59.116293 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.115520 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd5eb8b-a684-468d-b277-b07bd931e1e1" containerName="extract" Nov 02 02:51:59.117158 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.117137 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.209808 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.209781 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 02:51:59.209939 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.209817 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 02:51:59.219194 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.219175 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 02:51:59.219307 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.219182 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 02:51:59.219307 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.219267 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2r6bp\"" Nov 02 02:51:59.219307 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.219303 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 02:51:59.242503 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.242481 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.242618 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.242511 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrsqj\" (UniqueName: \"kubernetes.io/projected/63246a3a-5a01-4ee6-b03b-855e09dd1e38-kube-api-access-xrsqj\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.242618 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.242549 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63246a3a-5a01-4ee6-b03b-855e09dd1e38-manifests\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.242689 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.242628 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.242723 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.242699 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63246a3a-5a01-4ee6-b03b-855e09dd1e38-tmp\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.242756 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.242733 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.343837 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.343808 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.343961 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.343852 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.343961 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.343870 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsqj\" (UniqueName: \"kubernetes.io/projected/63246a3a-5a01-4ee6-b03b-855e09dd1e38-kube-api-access-xrsqj\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.343961 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.343905 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63246a3a-5a01-4ee6-b03b-855e09dd1e38-manifests\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.343961 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.343924 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.343961 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.343939 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63246a3a-5a01-4ee6-b03b-855e09dd1e38-tmp\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.344437 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.344409 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63246a3a-5a01-4ee6-b03b-855e09dd1e38-manifests\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.346257 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.346234 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63246a3a-5a01-4ee6-b03b-855e09dd1e38-tmp\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.346459 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.346434 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.346530 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.346439 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.346571 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.346539 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63246a3a-5a01-4ee6-b03b-855e09dd1e38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.359272 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.359247 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b"] Nov 02 02:51:59.401488 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.401435 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrsqj\" (UniqueName: \"kubernetes.io/projected/63246a3a-5a01-4ee6-b03b-855e09dd1e38-kube-api-access-xrsqj\") pod \"opendatahub-operator-controller-manager-5b55749cb7-lm22b\" (UID: \"63246a3a-5a01-4ee6-b03b-855e09dd1e38\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.429059 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.429040 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:51:59.907108 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:51:59.907083 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b"] Nov 02 02:51:59.909555 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:51:59.909533 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63246a3a_5a01_4ee6_b03b_855e09dd1e38.slice/crio-cae4c489c0d1072917d8fe30e9fc3ce3ce2c87943525274ab0d4c3504a6f2234 WatchSource:0}: Error finding container cae4c489c0d1072917d8fe30e9fc3ce3ce2c87943525274ab0d4c3504a6f2234: Status 404 returned error can't find the container with id cae4c489c0d1072917d8fe30e9fc3ce3ce2c87943525274ab0d4c3504a6f2234 Nov 02 02:52:00.227591 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:00.227514 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" event={"ID":"63246a3a-5a01-4ee6-b03b-855e09dd1e38","Type":"ContainerStarted","Data":"cae4c489c0d1072917d8fe30e9fc3ce3ce2c87943525274ab0d4c3504a6f2234"} Nov 02 02:52:03.450609 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:52:03.450570 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:52:04.245477 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:04.245399 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" event={"ID":"63246a3a-5a01-4ee6-b03b-855e09dd1e38","Type":"ContainerStarted","Data":"a378ab65a75262f69aaf8fbc9716e940584f8d2d0eac87730e9813cf4704012a"} Nov 02 02:52:05.250601 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:05.250566 4868 generic.go:356] "Generic (PLEG): container finished" podID="63246a3a-5a01-4ee6-b03b-855e09dd1e38" containerID="a378ab65a75262f69aaf8fbc9716e940584f8d2d0eac87730e9813cf4704012a" exitCode=0 Nov 02 02:52:05.251062 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:05.250648 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" event={"ID":"63246a3a-5a01-4ee6-b03b-855e09dd1e38","Type":"ContainerDied","Data":"a378ab65a75262f69aaf8fbc9716e940584f8d2d0eac87730e9813cf4704012a"} Nov 02 02:52:07.260762 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:07.260720 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" event={"ID":"63246a3a-5a01-4ee6-b03b-855e09dd1e38","Type":"ContainerStarted","Data":"9803afc87ddacd653887e46a2366280b447c2baf03ba0898f969285f58c8ee2d"} Nov 02 02:52:07.261157 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:07.260836 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:52:07.537026 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:07.536929 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" podStartSLOduration=4.050214685 podStartE2EDuration="9.536914817s" totalImagesPullingTime="5.017338872s" totalInitContainerRuntime="469.36126ms" isStatefulPod=false podCreationTimestamp="2026-11-02 02:51:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:07.486919803 +0000 UTC m=+269.482620745" watchObservedRunningTime="2026-11-02 02:52:07.536914817 +0000 UTC m=+269.532615760" Nov 02 02:52:18.267576 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:18.267550 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b55749cb7-lm22b" Nov 02 02:52:31.293344 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.293307 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2"] Nov 02 02:52:31.295384 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.295366 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.361191 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.361157 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 02:52:31.379637 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.379609 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6b08ff-10f3-4655-b67b-18931ee4162a-tls-certs\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.379745 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.379651 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsbh\" (UniqueName: \"kubernetes.io/projected/8e6b08ff-10f3-4655-b67b-18931ee4162a-kube-api-access-4hsbh\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.379789 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.379756 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e6b08ff-10f3-4655-b67b-18931ee4162a-tmp\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.396204 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.396175 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 02:52:31.396204 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.396183 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cksvz\"" Nov 02 02:52:31.397509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.397489 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2"] Nov 02 02:52:31.480469 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.480445 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6b08ff-10f3-4655-b67b-18931ee4162a-tls-certs\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.480578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.480489 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsbh\" (UniqueName: \"kubernetes.io/projected/8e6b08ff-10f3-4655-b67b-18931ee4162a-kube-api-access-4hsbh\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.480578 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.480528 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e6b08ff-10f3-4655-b67b-18931ee4162a-tmp\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.482776 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.482746 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e6b08ff-10f3-4655-b67b-18931ee4162a-tmp\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.482975 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.482960 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6b08ff-10f3-4655-b67b-18931ee4162a-tls-certs\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.536788 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.536762 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsbh\" (UniqueName: \"kubernetes.io/projected/8e6b08ff-10f3-4655-b67b-18931ee4162a-kube-api-access-4hsbh\") pod \"kube-auth-proxy-66599bc4bb-z7fp2\" (UID: \"8e6b08ff-10f3-4655-b67b-18931ee4162a\") " pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.604767 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.604742 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" Nov 02 02:52:31.970017 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:31.969935 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2"] Nov 02 02:52:31.972951 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:52:31.972922 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6b08ff_10f3_4655_b67b_18931ee4162a.slice/crio-9b5d58a6e7380171fb877512241a6069d2c6ec9a7f4a4e4fa8e147a8a58156e0 WatchSource:0}: Error finding container 9b5d58a6e7380171fb877512241a6069d2c6ec9a7f4a4e4fa8e147a8a58156e0: Status 404 returned error can't find the container with id 9b5d58a6e7380171fb877512241a6069d2c6ec9a7f4a4e4fa8e147a8a58156e0 Nov 02 02:52:32.365259 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:32.365231 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" event={"ID":"8e6b08ff-10f3-4655-b67b-18931ee4162a","Type":"ContainerStarted","Data":"9b5d58a6e7380171fb877512241a6069d2c6ec9a7f4a4e4fa8e147a8a58156e0"} Nov 02 02:52:33.449021 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:52:33.448955 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:52:36.384689 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:36.384648 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" event={"ID":"8e6b08ff-10f3-4655-b67b-18931ee4162a","Type":"ContainerStarted","Data":"4d9f72e29343205b9b5a1653e85e8b288ac5956d61fe39270c62f625f8986694"} Nov 02 02:52:36.499502 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:36.499432 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66599bc4bb-z7fp2" podStartSLOduration=1.854246076 podStartE2EDuration="5.499416876s" totalImagesPullingTime="3.6451708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:52:36.498836775 +0000 UTC m=+298.494537721" watchObservedRunningTime="2026-11-02 02:52:36.499416876 +0000 UTC m=+298.495117819" Nov 02 02:52:38.131895 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:38.131867 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:52:38.132646 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:52:38.132623 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:53:03.447016 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:53:03.446928 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:53:33.447136 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:53:33.447097 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:53:42.774467 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.774430 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w"] Nov 02 02:53:42.777994 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.777973 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.880333 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.880304 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w"] Nov 02 02:53:42.888874 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.888853 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.888984 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.888882 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.888984 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.888915 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.889064 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.889038 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.889107 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.889085 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5rg\" (UniqueName: \"kubernetes.io/projected/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-kube-api-access-8s5rg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.889165 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.889148 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.889248 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.889187 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.989825 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.989794 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5rg\" (UniqueName: \"kubernetes.io/projected/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-kube-api-access-8s5rg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.989968 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.989837 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.989968 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.989861 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.989968 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.989897 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.989968 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.989924 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.989968 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.989965 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.990238 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.990095 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.990706 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.990678 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.992320 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.992293 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.992436 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.992327 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.992436 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.992390 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:42.992509 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:42.992467 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:43.048042 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.047985 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5rg\" (UniqueName: \"kubernetes.io/projected/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-kube-api-access-8s5rg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:43.049237 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.049218 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lqm9w\" (UID: \"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:43.089304 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.089282 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:43.510487 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.510461 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w"] Nov 02 02:53:43.512318 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:53:43.512290 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2eaf6cc_9d41_43cb_98e2_1a5d910fa4f9.slice/crio-7bacc475f5684306c3cc736d695c8a3534acfba9ff05e7b139eddfbd9f88a06f WatchSource:0}: Error finding container 7bacc475f5684306c3cc736d695c8a3534acfba9ff05e7b139eddfbd9f88a06f: Status 404 returned error can't find the container with id 7bacc475f5684306c3cc736d695c8a3534acfba9ff05e7b139eddfbd9f88a06f Nov 02 02:53:43.514343 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.514312 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:53:43.514431 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.514392 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:53:43.652780 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.652743 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" event={"ID":"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9","Type":"ContainerStarted","Data":"c6eff6b4471adcdb8c0853d3c2bab23f477c66dd438c08394c937993208b9f63"} Nov 02 02:53:43.652959 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.652791 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" event={"ID":"e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9","Type":"ContainerStarted","Data":"7bacc475f5684306c3cc736d695c8a3534acfba9ff05e7b139eddfbd9f88a06f"} Nov 02 02:53:43.652959 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.652876 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:43.770331 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:43.770236 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" podStartSLOduration=1.770221357 podStartE2EDuration="1.770221357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:53:43.76915437 +0000 UTC m=+365.764855317" watchObservedRunningTime="2026-11-02 02:53:43.770221357 +0000 UTC m=+365.765922300" Nov 02 02:53:44.658397 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:44.658337 4868 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lqm9w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:53:44.658769 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:44.658415 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" podUID="e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:53:47.657510 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:47.657478 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lqm9w" Nov 02 02:53:47.874727 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:47.874689 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j"] Nov 02 02:53:47.874939 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:47.874916 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerName="discovery" containerID="cri-o://1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620" gracePeriod=30 Nov 02 02:53:48.118712 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.118684 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:53:48.128138 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128105 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.128271 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128171 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.128271 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128203 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.128271 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128230 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.128271 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128254 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.128500 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128427 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.128500 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.128467 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap\") pod \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\" (UID: \"6759b1eb-54b3-4ad2-90eb-6e9430a94b31\") " Nov 02 02:53:48.129449 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.129308 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:53:48.130810 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.130787 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:53:48.131314 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.131270 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:53:48.131314 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.131270 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "kube-api-access-d9hhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:53:48.133281 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.133248 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:53:48.133416 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.133328 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:53:48.133581 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.133566 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig" pod "6759b1eb-54b3-4ad2-90eb-6e9430a94b31" (UID: "6759b1eb-54b3-4ad2-90eb-6e9430a94b31"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:53:48.229693 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229621 4868 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-cacerts\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.229693 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229650 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9hhs\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-kube-api-access-d9hhs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.229693 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229669 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.229693 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229682 4868 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-local-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.229896 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229697 4868 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-ca-configmap\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.229896 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229711 4868 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-kubeconfig\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.229896 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.229723 4868 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6759b1eb-54b3-4ad2-90eb-6e9430a94b31-istio-csr-dns-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:53:48.676428 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.676383 4868 generic.go:356] "Generic (PLEG): container finished" podID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerID="1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620" exitCode=0 Nov 02 02:53:48.676882 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.676503 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" Nov 02 02:53:48.676882 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.676533 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" event={"ID":"6759b1eb-54b3-4ad2-90eb-6e9430a94b31","Type":"ContainerDied","Data":"1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620"} Nov 02 02:53:48.676882 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.676600 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j" event={"ID":"6759b1eb-54b3-4ad2-90eb-6e9430a94b31","Type":"ContainerDied","Data":"42fcdcc7f6f1b1844000ec92164b2a99218f162caa1990155536fc4edca554aa"} Nov 02 02:53:48.676882 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.676633 4868 scope.go:122] "RemoveContainer" containerID="1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620" Nov 02 02:53:48.686815 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.686795 4868 scope.go:122] "RemoveContainer" containerID="1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620" Nov 02 02:53:48.687296 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:53:48.687173 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620\": container with ID starting with 1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620 not found: ID does not exist" containerID="1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620" Nov 02 02:53:48.687296 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.687208 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620"} err="failed to get container status \"1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620\": rpc error: code = NotFound desc = could not find container \"1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620\": container with ID starting with 1876342405b7a169d6da7e166aa00dab6766ca25d21f937f447ab2c316bfc620 not found: ID does not exist" Nov 02 02:53:48.846744 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.846718 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j"] Nov 02 02:53:48.852847 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:48.852811 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kx7j"] Nov 02 02:53:49.046898 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.046821 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9"] Nov 02 02:53:49.047231 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.047213 4868 cpu_manager.go:414] "containerMap: removing container" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerName="discovery" Nov 02 02:53:49.047316 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.047233 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerName="discovery" Nov 02 02:53:49.047398 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.047332 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" containerName="discovery" Nov 02 02:53:49.051305 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.051285 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.105960 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.105935 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 02:53:49.106092 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.105986 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 02:53:49.106092 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.106079 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 02:53:49.106233 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.106100 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sxzw4\"" Nov 02 02:53:49.107929 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.107909 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9"] Nov 02 02:53:49.137925 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.137901 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r9zv9\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.138026 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.137953 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzxp\" (UniqueName: \"kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp\") pod \"llmisvc-controller-manager-869d5dcc5d-r9zv9\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.238870 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.238850 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzxp\" (UniqueName: \"kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp\") pod \"llmisvc-controller-manager-869d5dcc5d-r9zv9\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.238974 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.238897 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r9zv9\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.241177 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.241160 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r9zv9\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.294892 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.294869 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzxp\" (UniqueName: \"kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp\") pod \"llmisvc-controller-manager-869d5dcc5d-r9zv9\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.362935 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.362911 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:49.729046 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.728938 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9"] Nov 02 02:53:49.731496 ip-10-0-137-74 kubenswrapper[4868]: W1102 02:53:49.731467 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod93a941f7_6862_4602_a6dd_8516484742b8.slice/crio-c51b398e0b90804385bd446b384acbba2211e769fbe9aa618d374d24460e90dd WatchSource:0}: Error finding container c51b398e0b90804385bd446b384acbba2211e769fbe9aa618d374d24460e90dd: Status 404 returned error can't find the container with id c51b398e0b90804385bd446b384acbba2211e769fbe9aa618d374d24460e90dd Nov 02 02:53:49.732829 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:49.732808 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 02:53:50.112419 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:50.112391 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6759b1eb-54b3-4ad2-90eb-6e9430a94b31" path="/var/lib/kubelet/pods/6759b1eb-54b3-4ad2-90eb-6e9430a94b31/volumes" Nov 02 02:53:50.688060 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:50.688025 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" event={"ID":"93a941f7-6862-4602-a6dd-8516484742b8","Type":"ContainerStarted","Data":"c51b398e0b90804385bd446b384acbba2211e769fbe9aa618d374d24460e90dd"} Nov 02 02:53:52.699363 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:52.699314 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" event={"ID":"93a941f7-6862-4602-a6dd-8516484742b8","Type":"ContainerStarted","Data":"160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951"} Nov 02 02:53:52.699734 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:52.699398 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:53:52.813902 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:53:52.813855 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" podStartSLOduration=1.740720682 podStartE2EDuration="3.813842199s" totalImagesPullingTime="2.073121517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:53:52.812037574 +0000 UTC m=+374.807738517" watchObservedRunningTime="2026-11-02 02:53:52.813842199 +0000 UTC m=+374.809543157" Nov 02 02:54:03.448142 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:54:03.448094 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:54:23.705504 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:54:23.705427 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 02:54:33.446921 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:54:33.446884 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:55:03.445414 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:55:03.445371 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:55:33.448149 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:55:33.448105 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:56:03.445282 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:56:03.445194 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:56:33.445700 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:56:33.445655 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:57:03.446849 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:57:03.446808 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:57:33.446903 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:57:33.446860 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:57:38.160206 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:57:38.160175 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:57:38.162655 ip-10-0-137-74 kubenswrapper[4868]: I1102 02:57:38.162631 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 02:58:03.444273 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:58:03.444231 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:58:33.446200 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:58:33.446160 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:59:03.447838 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:59:03.447749 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 02:59:33.445648 ip-10-0-137-74 kubenswrapper[4868]: E1102 02:59:33.445606 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:00:03.445550 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:00:03.445510 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:00:33.445544 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:00:33.445497 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:01:03.445462 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:01:03.445425 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:01:33.444637 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:01:33.444600 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:02:03.445212 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:02:03.445174 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:02:33.445983 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:02:33.445944 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:02:38.192557 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:02:38.192529 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:02:38.195236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:02:38.195213 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:03:03.445459 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:03:03.445423 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:03:09.877135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.877102 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj"] Nov 02 03:03:09.880195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.880179 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:09.885261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.885232 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:09.888274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.888257 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:09.933626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.933586 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 03:03:09.933762 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.933627 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5fwnq\"" Nov 02 03:03:09.939659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.939638 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 03:03:09.948854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.948838 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 03:03:09.948854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.948841 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 03:03:09.954759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.954736 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj"] Nov 02 03:03:09.955732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:09.955710 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:10.010092 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.010055 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file\") pod \"limitador-limitador-74dfcfb894-p57jd\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.011728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.010639 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.011944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.011921 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twh5r\" (UniqueName: \"kubernetes.io/projected/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-kube-api-access-twh5r\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.012134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.012119 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.012250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.012235 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsrs\" (UniqueName: \"kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs\") pod \"limitador-limitador-74dfcfb894-p57jd\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.039830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.039803 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 03:03:10.112656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.112632 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.112788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.112662 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsrs\" (UniqueName: \"kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs\") pod \"limitador-limitador-74dfcfb894-p57jd\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.112788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.112691 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file\") pod \"limitador-limitador-74dfcfb894-p57jd\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.113002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.112981 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.113076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.113045 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twh5r\" (UniqueName: \"kubernetes.io/projected/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-kube-api-access-twh5r\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.113479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.113455 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file\") pod \"limitador-limitador-74dfcfb894-p57jd\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.113479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.113473 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.115483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.115458 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.148789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.148734 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:10.170205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.170176 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsrs\" (UniqueName: \"kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs\") pod \"limitador-limitador-74dfcfb894-p57jd\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.170296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.170281 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twh5r\" (UniqueName: \"kubernetes.io/projected/cce70044-e2d2-40aa-a55b-f1ba8f1cc976-kube-api-access-twh5r\") pod \"kuadrant-console-plugin-66c4fb575d-84qbj\" (UID: \"cce70044-e2d2-40aa-a55b-f1ba8f1cc976\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.204239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.204214 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" Nov 02 03:03:10.215575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.215554 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:10.579335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.579309 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj"] Nov 02 03:03:10.582016 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:03:10.581980 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcce70044_e2d2_40aa_a55b_f1ba8f1cc976.slice/crio-85b786d59294c4ce958ee5f7d6fac7ad5cba7954a5ec9bd4f9359af885b0a873 WatchSource:0}: Error finding container 85b786d59294c4ce958ee5f7d6fac7ad5cba7954a5ec9bd4f9359af885b0a873: Status 404 returned error can't find the container with id 85b786d59294c4ce958ee5f7d6fac7ad5cba7954a5ec9bd4f9359af885b0a873 Nov 02 03:03:10.583305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.583286 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:03:10.629726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:10.629702 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:10.631931 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:03:10.631901 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd1550a_1f2d_4806_9070_b5e83ca24c7e.slice/crio-b2366b3e37d1c5b071d3b3eadbdc757045ba5bccc2562748afecc29762120969 WatchSource:0}: Error finding container b2366b3e37d1c5b071d3b3eadbdc757045ba5bccc2562748afecc29762120969: Status 404 returned error can't find the container with id b2366b3e37d1c5b071d3b3eadbdc757045ba5bccc2562748afecc29762120969 Nov 02 03:03:11.009456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:11.009422 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" event={"ID":"3bd1550a-1f2d-4806-9070-b5e83ca24c7e","Type":"ContainerStarted","Data":"b2366b3e37d1c5b071d3b3eadbdc757045ba5bccc2562748afecc29762120969"} Nov 02 03:03:11.010641 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:11.010620 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" event={"ID":"cce70044-e2d2-40aa-a55b-f1ba8f1cc976","Type":"ContainerStarted","Data":"85b786d59294c4ce958ee5f7d6fac7ad5cba7954a5ec9bd4f9359af885b0a873"} Nov 02 03:03:15.033127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:15.033089 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" event={"ID":"3bd1550a-1f2d-4806-9070-b5e83ca24c7e","Type":"ContainerStarted","Data":"45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e"} Nov 02 03:03:15.033636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:15.033235 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:15.148568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:15.148514 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" podStartSLOduration=2.307956595 podStartE2EDuration="6.148496924s" totalImagesPullingTime="3.840540329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:03:15.147249418 +0000 UTC m=+937.142950361" watchObservedRunningTime="2026-11-02 03:03:15.148496924 +0000 UTC m=+937.144197868" Nov 02 03:03:18.047961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:18.047922 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" event={"ID":"cce70044-e2d2-40aa-a55b-f1ba8f1cc976","Type":"ContainerStarted","Data":"ee3c535d03b6bf0764b4e7222249d9e0e8ecd19e570516d0787428c510bb03db"} Nov 02 03:03:18.162527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:18.162471 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-84qbj" podStartSLOduration=2.649773125 podStartE2EDuration="9.162457957s" totalImagesPullingTime="6.512684832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:03:18.161852945 +0000 UTC m=+940.157553903" watchObservedRunningTime="2026-11-02 03:03:18.162457957 +0000 UTC m=+940.158158898" Nov 02 03:03:26.038803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:26.038774 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:27.222587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.222554 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:27.223036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.222723 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" podUID="3bd1550a-1f2d-4806-9070-b5e83ca24c7e" containerName="limitador" containerID="cri-o://45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e" gracePeriod=30 Nov 02 03:03:27.761010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.760986 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:27.848344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.848284 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file\" (UniqueName: \"kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file\") pod \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " Nov 02 03:03:27.848456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.848408 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs\" (UniqueName: \"kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs\") pod \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\" (UID: \"3bd1550a-1f2d-4806-9070-b5e83ca24c7e\") " Nov 02 03:03:27.848685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.848666 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file" pod "3bd1550a-1f2d-4806-9070-b5e83ca24c7e" (UID: "3bd1550a-1f2d-4806-9070-b5e83ca24c7e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:03:27.850561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.850538 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs" pod "3bd1550a-1f2d-4806-9070-b5e83ca24c7e" (UID: "3bd1550a-1f2d-4806-9070-b5e83ca24c7e"). InnerVolumeSpecName "kube-api-access-9bsrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:03:27.949364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.949324 4868 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-config-file\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:03:27.949480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:27.949347 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsrs\" (UniqueName: \"kubernetes.io/projected/3bd1550a-1f2d-4806-9070-b5e83ca24c7e-kube-api-access-9bsrs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:03:28.090464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.090436 4868 generic.go:356] "Generic (PLEG): container finished" podID="3bd1550a-1f2d-4806-9070-b5e83ca24c7e" containerID="45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e" exitCode=0 Nov 02 03:03:28.090581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.090481 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" Nov 02 03:03:28.090581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.090519 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" event={"ID":"3bd1550a-1f2d-4806-9070-b5e83ca24c7e","Type":"ContainerDied","Data":"45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e"} Nov 02 03:03:28.090581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.090571 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-p57jd" event={"ID":"3bd1550a-1f2d-4806-9070-b5e83ca24c7e","Type":"ContainerDied","Data":"b2366b3e37d1c5b071d3b3eadbdc757045ba5bccc2562748afecc29762120969"} Nov 02 03:03:28.090738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.090593 4868 scope.go:122] "RemoveContainer" containerID="45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e" Nov 02 03:03:28.099884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.099868 4868 scope.go:122] "RemoveContainer" containerID="45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e" Nov 02 03:03:28.100134 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:03:28.100117 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e\": container with ID starting with 45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e not found: ID does not exist" containerID="45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e" Nov 02 03:03:28.100203 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.100144 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e"} err="failed to get container status \"45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e\": rpc error: code = NotFound desc = could not find container \"45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e\": container with ID starting with 45189722e1a25405f3e938aabcedd36ef096a732b81f0a317ed79339d96e3c4e not found: ID does not exist" Nov 02 03:03:28.254409 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.254378 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:28.303824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:28.303790 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-p57jd"] Nov 02 03:03:30.117682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:30.117626 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd1550a-1f2d-4806-9070-b5e83ca24c7e" path="/var/lib/kubelet/pods/3bd1550a-1f2d-4806-9070-b5e83ca24c7e/volumes" Nov 02 03:03:33.444445 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:03:33.444406 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:03:33.905394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.905361 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8"] Nov 02 03:03:33.905716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.905700 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3bd1550a-1f2d-4806-9070-b5e83ca24c7e" containerName="limitador" Nov 02 03:03:33.905761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.905718 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd1550a-1f2d-4806-9070-b5e83ca24c7e" containerName="limitador" Nov 02 03:03:33.905795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.905785 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd1550a-1f2d-4806-9070-b5e83ca24c7e" containerName="limitador" Nov 02 03:03:33.909913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.909897 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:33.971105 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.971076 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8"] Nov 02 03:03:33.996748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.996720 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/401ad747-9fed-4a52-a560-0bab16515219-data\") pod \"postgres-7cdbf6dc44-pgtf8\" (UID: \"401ad747-9fed-4a52-a560-0bab16515219\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:33.996877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:33.996757 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsqx\" (UniqueName: \"kubernetes.io/projected/401ad747-9fed-4a52-a560-0bab16515219-kube-api-access-nhsqx\") pod \"postgres-7cdbf6dc44-pgtf8\" (UID: \"401ad747-9fed-4a52-a560-0bab16515219\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:34.010543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.010523 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 03:03:34.018023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.018001 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 03:03:34.042188 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.042169 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 03:03:34.060194 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.060176 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bjvl6\"" Nov 02 03:03:34.098054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.098026 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/401ad747-9fed-4a52-a560-0bab16515219-data\") pod \"postgres-7cdbf6dc44-pgtf8\" (UID: \"401ad747-9fed-4a52-a560-0bab16515219\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:34.098137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.098066 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsqx\" (UniqueName: \"kubernetes.io/projected/401ad747-9fed-4a52-a560-0bab16515219-kube-api-access-nhsqx\") pod \"postgres-7cdbf6dc44-pgtf8\" (UID: \"401ad747-9fed-4a52-a560-0bab16515219\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:34.098498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.098483 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/401ad747-9fed-4a52-a560-0bab16515219-data\") pod \"postgres-7cdbf6dc44-pgtf8\" (UID: \"401ad747-9fed-4a52-a560-0bab16515219\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:34.154995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.154970 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsqx\" (UniqueName: \"kubernetes.io/projected/401ad747-9fed-4a52-a560-0bab16515219-kube-api-access-nhsqx\") pod \"postgres-7cdbf6dc44-pgtf8\" (UID: \"401ad747-9fed-4a52-a560-0bab16515219\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:34.221590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.221536 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:34.588250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:34.588199 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8"] Nov 02 03:03:34.590547 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:03:34.590518 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401ad747_9fed_4a52_a560_0bab16515219.slice/crio-e9072e3c23874d557fd4f55559c83a50e3e941fd931a452ca577b9a415450db6 WatchSource:0}: Error finding container e9072e3c23874d557fd4f55559c83a50e3e941fd931a452ca577b9a415450db6: Status 404 returned error can't find the container with id e9072e3c23874d557fd4f55559c83a50e3e941fd931a452ca577b9a415450db6 Nov 02 03:03:35.121478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:35.121441 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" event={"ID":"401ad747-9fed-4a52-a560-0bab16515219","Type":"ContainerStarted","Data":"e9072e3c23874d557fd4f55559c83a50e3e941fd931a452ca577b9a415450db6"} Nov 02 03:03:41.449975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:41.449949 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 03:03:42.153665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:42.153629 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" event={"ID":"401ad747-9fed-4a52-a560-0bab16515219","Type":"ContainerStarted","Data":"5e37c734d9aad901778a080622bfff75f1f440071e22af5363ccb785c465036d"} Nov 02 03:03:42.153854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:42.153761 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:42.268758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:42.268711 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" podStartSLOduration=2.4623451 podStartE2EDuration="9.2686986s" totalImagesPullingTime="6.8063535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:03:42.267185709 +0000 UTC m=+964.262886664" watchObservedRunningTime="2026-11-02 03:03:42.2686986 +0000 UTC m=+964.264399540" Nov 02 03:03:48.185456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:48.185424 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgtf8" Nov 02 03:03:54.896327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.896293 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57f6c8c568-lb862"] Nov 02 03:03:54.899435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.899418 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:54.951943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.951914 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 03:03:54.951943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.951914 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 03:03:54.951943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.951920 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-scz2n\"" Nov 02 03:03:54.958407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.958380 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:54.958542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.958417 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:54.958605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:54.958563 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f95c\" (UniqueName: \"kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.002406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.002378 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 03:03:55.002728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.002709 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57f6c8c568-lb862"] Nov 02 03:03:55.059211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.059176 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.059211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.059208 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.059388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.059259 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f95c\" (UniqueName: \"kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.059388 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:03:55.059336 4868 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Nov 02 03:03:55.059464 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:03:55.059431 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs podName:87648601-8705-4a74-b72e-055382b6d678 nodeName:}" failed. No retries permitted until 2026-11-02 03:03:55.559412294 +0000 UTC m=+977.555113215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs") pod "maas-controller-57f6c8c568-lb862" (UID: "87648601-8705-4a74-b72e-055382b6d678") : secret "maas-controller-metrics-tls" not found Nov 02 03:03:55.061776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.061755 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.116814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.116789 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f95c\" (UniqueName: \"kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.443634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.443603 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dbf54fd5c-ttkx8"] Nov 02 03:03:55.446934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.446918 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.461701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.461677 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqck\" (UniqueName: \"kubernetes.io/projected/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-kube-api-access-ltqck\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.461802 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.461714 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-webhook-cert\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.461842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.461813 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-metrics-certs\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.510912 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.510888 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf54fd5c-ttkx8"] Nov 02 03:03:55.562316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.562287 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqck\" (UniqueName: \"kubernetes.io/projected/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-kube-api-access-ltqck\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.562469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.562339 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-webhook-cert\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.562469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.562405 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.562469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.562435 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-metrics-certs\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.565014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.564986 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\") pod \"maas-controller-57f6c8c568-lb862\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:55.565014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.565000 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-webhook-cert\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.565139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.565061 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-metrics-certs\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.619528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.619501 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqck\" (UniqueName: \"kubernetes.io/projected/f5e7e1a3-15ef-485e-a84e-a257d62e8f14-kube-api-access-ltqck\") pod \"maas-controller-7dbf54fd5c-ttkx8\" (UID: \"f5e7e1a3-15ef-485e-a84e-a257d62e8f14\") " pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.759430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.759368 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:03:55.811538 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:55.811509 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:03:56.127676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:56.127642 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dbf54fd5c-ttkx8"] Nov 02 03:03:56.159027 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:03:56.158993 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e7e1a3_15ef_485e_a84e_a257d62e8f14.slice/crio-090edb953161b5805327bbf7f4e0fa39c465b441c8cba7d18e2307d3025ab958 WatchSource:0}: Error finding container 090edb953161b5805327bbf7f4e0fa39c465b441c8cba7d18e2307d3025ab958: Status 404 returned error can't find the container with id 090edb953161b5805327bbf7f4e0fa39c465b441c8cba7d18e2307d3025ab958 Nov 02 03:03:56.207783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:56.207753 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" event={"ID":"f5e7e1a3-15ef-485e-a84e-a257d62e8f14","Type":"ContainerStarted","Data":"090edb953161b5805327bbf7f4e0fa39c465b441c8cba7d18e2307d3025ab958"} Nov 02 03:03:56.237229 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:56.237209 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57f6c8c568-lb862"] Nov 02 03:03:56.239295 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:03:56.239270 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87648601_8705_4a74_b72e_055382b6d678.slice/crio-0a75fcb6e2338bebec1cbca6c675850deb8a539777073f983934ac787c57045f WatchSource:0}: Error finding container 0a75fcb6e2338bebec1cbca6c675850deb8a539777073f983934ac787c57045f: Status 404 returned error can't find the container with id 0a75fcb6e2338bebec1cbca6c675850deb8a539777073f983934ac787c57045f Nov 02 03:03:57.212729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:03:57.212698 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f6c8c568-lb862" event={"ID":"87648601-8705-4a74-b72e-055382b6d678","Type":"ContainerStarted","Data":"0a75fcb6e2338bebec1cbca6c675850deb8a539777073f983934ac787c57045f"} Nov 02 03:04:03.243484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:03.243452 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f6c8c568-lb862" event={"ID":"87648601-8705-4a74-b72e-055382b6d678","Type":"ContainerStarted","Data":"28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c"} Nov 02 03:04:03.243837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:03.243566 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:04:03.386601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:03.386513 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57f6c8c568-lb862" podStartSLOduration=2.502483634 podStartE2EDuration="9.386494659s" totalImagesPullingTime="6.884011025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:04:03.385118627 +0000 UTC m=+985.380819573" watchObservedRunningTime="2026-11-02 03:04:03.386494659 +0000 UTC m=+985.382195603" Nov 02 03:04:03.444382 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:04:03.444320 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:04:04.248879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:04.248839 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" event={"ID":"f5e7e1a3-15ef-485e-a84e-a257d62e8f14","Type":"ContainerStarted","Data":"3d5948e1e47a41eec673c0272978836c93cbf606c50414ab43fb414d6b2b3a62"} Nov 02 03:04:04.249287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:04.249133 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:04:04.365861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:04.365808 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" podStartSLOduration=2.299159126 podStartE2EDuration="9.365792734s" totalImagesPullingTime="7.066633608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:04:04.364805635 +0000 UTC m=+986.360506579" watchObservedRunningTime="2026-11-02 03:04:04.365792734 +0000 UTC m=+986.361493676" Nov 02 03:04:14.251252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:14.251217 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:04:15.255192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.255160 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dbf54fd5c-ttkx8" Nov 02 03:04:15.422045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.422011 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57f6c8c568-lb862"] Nov 02 03:04:15.422281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.422250 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-57f6c8c568-lb862" podUID="87648601-8705-4a74-b72e-055382b6d678" containerName="manager" containerID="cri-o://28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c" gracePeriod=30 Nov 02 03:04:15.663489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.663467 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:04:15.718968 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.718946 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert\") pod \"87648601-8705-4a74-b72e-055382b6d678\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " Nov 02 03:04:15.719089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.718999 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c\" (UniqueName: \"kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c\") pod \"87648601-8705-4a74-b72e-055382b6d678\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " Nov 02 03:04:15.719089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.719057 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\") pod \"87648601-8705-4a74-b72e-055382b6d678\" (UID: \"87648601-8705-4a74-b72e-055382b6d678\") " Nov 02 03:04:15.721040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.721015 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert" pod "87648601-8705-4a74-b72e-055382b6d678" (UID: "87648601-8705-4a74-b72e-055382b6d678"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:04:15.721119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.721077 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c" pod "87648601-8705-4a74-b72e-055382b6d678" (UID: "87648601-8705-4a74-b72e-055382b6d678"). InnerVolumeSpecName "kube-api-access-5f95c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:04:15.721119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.721090 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs" pod "87648601-8705-4a74-b72e-055382b6d678" (UID: "87648601-8705-4a74-b72e-055382b6d678"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:04:15.820645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.820564 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:04:15.820645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.820601 4868 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87648601-8705-4a74-b72e-055382b6d678-webhook-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:04:15.820645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:15.820615 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f95c\" (UniqueName: \"kubernetes.io/projected/87648601-8705-4a74-b72e-055382b6d678-kube-api-access-5f95c\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:04:16.299040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.299001 4868 generic.go:356] "Generic (PLEG): container finished" podID="87648601-8705-4a74-b72e-055382b6d678" containerID="28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c" exitCode=0 Nov 02 03:04:16.299489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.299064 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57f6c8c568-lb862" Nov 02 03:04:16.299489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.299099 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f6c8c568-lb862" event={"ID":"87648601-8705-4a74-b72e-055382b6d678","Type":"ContainerDied","Data":"28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c"} Nov 02 03:04:16.299489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.299182 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57f6c8c568-lb862" event={"ID":"87648601-8705-4a74-b72e-055382b6d678","Type":"ContainerDied","Data":"0a75fcb6e2338bebec1cbca6c675850deb8a539777073f983934ac787c57045f"} Nov 02 03:04:16.299489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.299205 4868 scope.go:122] "RemoveContainer" containerID="28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c" Nov 02 03:04:16.307682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.307664 4868 scope.go:122] "RemoveContainer" containerID="28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c" Nov 02 03:04:16.307971 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:04:16.307951 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c\": container with ID starting with 28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c not found: ID does not exist" containerID="28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c" Nov 02 03:04:16.308025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.307978 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c"} err="failed to get container status \"28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c\": rpc error: code = NotFound desc = could not find container \"28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c\": container with ID starting with 28c837c36253e037d1f58b9bfc5b07bd73fc73139b77f3a1a98cb50850d2ee6c not found: ID does not exist" Nov 02 03:04:16.460386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.460297 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57f6c8c568-lb862"] Nov 02 03:04:16.509546 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:16.509515 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57f6c8c568-lb862"] Nov 02 03:04:18.112732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:18.112699 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87648601-8705-4a74-b72e-055382b6d678" path="/var/lib/kubelet/pods/87648601-8705-4a74-b72e-055382b6d678/volumes" Nov 02 03:04:33.444464 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:04:33.444428 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:04:40.360782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.360745 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56965f5749-28k9l"] Nov 02 03:04:40.361164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.361039 4868 cpu_manager.go:414] "containerMap: removing container" podUID="87648601-8705-4a74-b72e-055382b6d678" containerName="manager" Nov 02 03:04:40.361164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.361049 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87648601-8705-4a74-b72e-055382b6d678" containerName="manager" Nov 02 03:04:40.361164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.361134 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="87648601-8705-4a74-b72e-055382b6d678" containerName="manager" Nov 02 03:04:40.364990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.364975 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.420931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.420901 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56965f5749-28k9l"] Nov 02 03:04:40.468499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.468476 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pz7wg\"" Nov 02 03:04:40.468499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.468476 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 03:04:40.468713 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.468509 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 03:04:40.509726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.509704 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.509859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.509732 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.509859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.509760 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.509980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.509956 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.610626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.610604 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.610741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.610639 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.610741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.610656 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.610741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.610685 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.611387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.611316 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.612976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.612957 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.613154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.613131 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.666840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.666818 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq\") pod \"maas-api-56965f5749-28k9l\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:40.675810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:40.675794 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:41.043933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:41.043904 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56965f5749-28k9l"] Nov 02 03:04:41.046266 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:04:41.046241 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea3ee7f_2891_4662_9735_7ee636f1eae2.slice/crio-1339828bc8dedb2beb7447d44013b35108c91612481ff957c41b80fc1b74c1b2 WatchSource:0}: Error finding container 1339828bc8dedb2beb7447d44013b35108c91612481ff957c41b80fc1b74c1b2: Status 404 returned error can't find the container with id 1339828bc8dedb2beb7447d44013b35108c91612481ff957c41b80fc1b74c1b2 Nov 02 03:04:41.407667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:41.407627 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" event={"ID":"2ea3ee7f-2891-4662-9735-7ee636f1eae2","Type":"ContainerStarted","Data":"1339828bc8dedb2beb7447d44013b35108c91612481ff957c41b80fc1b74c1b2"} Nov 02 03:04:48.437828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:48.437786 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" event={"ID":"2ea3ee7f-2891-4662-9735-7ee636f1eae2","Type":"ContainerStarted","Data":"41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc"} Nov 02 03:04:48.438281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:48.437884 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:04:48.553396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:48.553319 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" podStartSLOduration=1.5090927299999999 podStartE2EDuration="8.553301179s" totalImagesPullingTime="7.044208449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:04:48.551921963 +0000 UTC m=+1030.547622906" watchObservedRunningTime="2026-11-02 03:04:48.553301179 +0000 UTC m=+1030.549002138" Nov 02 03:04:54.448252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:04:54.448215 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:05:01.885439 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:01.885403 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6spg5"] Nov 02 03:05:01.887678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:01.887655 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:01.945745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:01.945717 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6spg5"] Nov 02 03:05:02.076868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.076831 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07d60c1-9802-4fa0-9a5b-85ec83152e37-cert\") pod \"llmisvc-controller-manager-6466c554d8-6spg5\" (UID: \"e07d60c1-9802-4fa0-9a5b-85ec83152e37\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.077045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.076891 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbfq\" (UniqueName: \"kubernetes.io/projected/e07d60c1-9802-4fa0-9a5b-85ec83152e37-kube-api-access-7qbfq\") pod \"llmisvc-controller-manager-6466c554d8-6spg5\" (UID: \"e07d60c1-9802-4fa0-9a5b-85ec83152e37\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.178240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.178146 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbfq\" (UniqueName: \"kubernetes.io/projected/e07d60c1-9802-4fa0-9a5b-85ec83152e37-kube-api-access-7qbfq\") pod \"llmisvc-controller-manager-6466c554d8-6spg5\" (UID: \"e07d60c1-9802-4fa0-9a5b-85ec83152e37\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.178240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.178209 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07d60c1-9802-4fa0-9a5b-85ec83152e37-cert\") pod \"llmisvc-controller-manager-6466c554d8-6spg5\" (UID: \"e07d60c1-9802-4fa0-9a5b-85ec83152e37\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.180659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.180631 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e07d60c1-9802-4fa0-9a5b-85ec83152e37-cert\") pod \"llmisvc-controller-manager-6466c554d8-6spg5\" (UID: \"e07d60c1-9802-4fa0-9a5b-85ec83152e37\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.235454 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.235425 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbfq\" (UniqueName: \"kubernetes.io/projected/e07d60c1-9802-4fa0-9a5b-85ec83152e37-kube-api-access-7qbfq\") pod \"llmisvc-controller-manager-6466c554d8-6spg5\" (UID: \"e07d60c1-9802-4fa0-9a5b-85ec83152e37\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.498091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.498008 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:02.863963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:02.863941 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6spg5"] Nov 02 03:05:02.866230 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:05:02.866201 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07d60c1_9802_4fa0_9a5b_85ec83152e37.slice/crio-563a125cc10a1524fbb37121cf7a0d20fff8382e3164db296a72ee48227200d8 WatchSource:0}: Error finding container 563a125cc10a1524fbb37121cf7a0d20fff8382e3164db296a72ee48227200d8: Status 404 returned error can't find the container with id 563a125cc10a1524fbb37121cf7a0d20fff8382e3164db296a72ee48227200d8 Nov 02 03:05:03.445164 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:05:03.445125 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:05:03.502986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:03.502950 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" event={"ID":"e07d60c1-9802-4fa0-9a5b-85ec83152e37","Type":"ContainerStarted","Data":"563a125cc10a1524fbb37121cf7a0d20fff8382e3164db296a72ee48227200d8"} Nov 02 03:05:04.508795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:04.508762 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" event={"ID":"e07d60c1-9802-4fa0-9a5b-85ec83152e37","Type":"ContainerStarted","Data":"2f5c09fab7b878015a6461e37f14b1480d45946aff291191bed63311c2c04b94"} Nov 02 03:05:04.509166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:04.508877 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:04.626095 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:04.626049 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" podStartSLOduration=2.853714744 podStartE2EDuration="3.626036858s" totalImagesPullingTime="772.322114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:05:04.62441845 +0000 UTC m=+1046.620119391" watchObservedRunningTime="2026-11-02 03:05:04.626036858 +0000 UTC m=+1046.621737801" Nov 02 03:05:05.659335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:05.659300 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56965f5749-28k9l"] Nov 02 03:05:05.659808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:05.659559 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" podUID="2ea3ee7f-2891-4662-9735-7ee636f1eae2" containerName="maas-api" containerID="cri-o://41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc" gracePeriod=30 Nov 02 03:05:05.894169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:05.894143 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:05:06.010933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.010855 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls\") pod \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " Nov 02 03:05:06.010933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.010904 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs\") pod \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " Nov 02 03:05:06.011156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.011010 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca\") pod \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " Nov 02 03:05:06.011156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.011051 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq\") pod \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\" (UID: \"2ea3ee7f-2891-4662-9735-7ee636f1eae2\") " Nov 02 03:05:06.011490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.011461 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca" pod "2ea3ee7f-2891-4662-9735-7ee636f1eae2" (UID: "2ea3ee7f-2891-4662-9735-7ee636f1eae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:05:06.013630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.013602 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq" pod "2ea3ee7f-2891-4662-9735-7ee636f1eae2" (UID: "2ea3ee7f-2891-4662-9735-7ee636f1eae2"). InnerVolumeSpecName "kube-api-access-7mjmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:05:06.013728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.013697 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls" pod "2ea3ee7f-2891-4662-9735-7ee636f1eae2" (UID: "2ea3ee7f-2891-4662-9735-7ee636f1eae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:05:06.013783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.013731 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs" pod "2ea3ee7f-2891-4662-9735-7ee636f1eae2" (UID: "2ea3ee7f-2891-4662-9735-7ee636f1eae2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:05:06.111590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.111562 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea3ee7f-2891-4662-9735-7ee636f1eae2-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:05:06.111590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.111590 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjmq\" (UniqueName: \"kubernetes.io/projected/2ea3ee7f-2891-4662-9735-7ee636f1eae2-kube-api-access-7mjmq\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:05:06.111771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.111605 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:05:06.111771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.111617 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ea3ee7f-2891-4662-9735-7ee636f1eae2-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:05:06.518517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.518485 4868 generic.go:356] "Generic (PLEG): container finished" podID="2ea3ee7f-2891-4662-9735-7ee636f1eae2" containerID="41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc" exitCode=0 Nov 02 03:05:06.518690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.518546 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" Nov 02 03:05:06.518690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.518557 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" event={"ID":"2ea3ee7f-2891-4662-9735-7ee636f1eae2","Type":"ContainerDied","Data":"41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc"} Nov 02 03:05:06.518690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.518595 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56965f5749-28k9l" event={"ID":"2ea3ee7f-2891-4662-9735-7ee636f1eae2","Type":"ContainerDied","Data":"1339828bc8dedb2beb7447d44013b35108c91612481ff957c41b80fc1b74c1b2"} Nov 02 03:05:06.518690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.518615 4868 scope.go:122] "RemoveContainer" containerID="41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc" Nov 02 03:05:06.526953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.526929 4868 scope.go:122] "RemoveContainer" containerID="41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc" Nov 02 03:05:06.527243 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:05:06.527222 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc\": container with ID starting with 41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc not found: ID does not exist" containerID="41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc" Nov 02 03:05:06.527289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.527254 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc"} err="failed to get container status \"41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc\": rpc error: code = NotFound desc = could not find container \"41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc\": container with ID starting with 41aae5d90bdbccc4d52d0d4753e7765bdec227d99a9014735ebb8a3ed91d42cc not found: ID does not exist" Nov 02 03:05:06.680371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.680334 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56965f5749-28k9l"] Nov 02 03:05:06.729557 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:06.729531 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56965f5749-28k9l"] Nov 02 03:05:08.113913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:08.113878 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea3ee7f-2891-4662-9735-7ee636f1eae2" path="/var/lib/kubelet/pods/2ea3ee7f-2891-4662-9735-7ee636f1eae2/volumes" Nov 02 03:05:33.444664 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:05:33.444624 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:05:35.515283 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:35.515249 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6spg5" Nov 02 03:05:35.685862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:35.685830 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9"] Nov 02 03:05:35.686097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:35.686069 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" podUID="93a941f7-6862-4602-a6dd-8516484742b8" containerName="manager" containerID="cri-o://160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951" gracePeriod=30 Nov 02 03:05:35.927687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:35.927664 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 03:05:36.032329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.032302 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert\" (UniqueName: \"kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert\") pod \"93a941f7-6862-4602-a6dd-8516484742b8\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " Nov 02 03:05:36.032474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.032376 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp\" (UniqueName: \"kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp\") pod \"93a941f7-6862-4602-a6dd-8516484742b8\" (UID: \"93a941f7-6862-4602-a6dd-8516484742b8\") " Nov 02 03:05:36.034318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.034290 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert" pod "93a941f7-6862-4602-a6dd-8516484742b8" (UID: "93a941f7-6862-4602-a6dd-8516484742b8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:05:36.034461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.034444 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp" pod "93a941f7-6862-4602-a6dd-8516484742b8" (UID: "93a941f7-6862-4602-a6dd-8516484742b8"). InnerVolumeSpecName "kube-api-access-dpzxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:05:36.133123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.133096 4868 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a941f7-6862-4602-a6dd-8516484742b8-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:05:36.133123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.133123 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzxp\" (UniqueName: \"kubernetes.io/projected/93a941f7-6862-4602-a6dd-8516484742b8-kube-api-access-dpzxp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:05:36.655595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.655564 4868 generic.go:356] "Generic (PLEG): container finished" podID="93a941f7-6862-4602-a6dd-8516484742b8" containerID="160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951" exitCode=0 Nov 02 03:05:36.656085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.655621 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" Nov 02 03:05:36.656085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.655654 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" event={"ID":"93a941f7-6862-4602-a6dd-8516484742b8","Type":"ContainerDied","Data":"160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951"} Nov 02 03:05:36.656085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.655713 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9" event={"ID":"93a941f7-6862-4602-a6dd-8516484742b8","Type":"ContainerDied","Data":"c51b398e0b90804385bd446b384acbba2211e769fbe9aa618d374d24460e90dd"} Nov 02 03:05:36.656085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.655736 4868 scope.go:122] "RemoveContainer" containerID="160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951" Nov 02 03:05:36.664456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.664436 4868 scope.go:122] "RemoveContainer" containerID="160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951" Nov 02 03:05:36.664782 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:05:36.664756 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951\": container with ID starting with 160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951 not found: ID does not exist" containerID="160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951" Nov 02 03:05:36.664881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.664790 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951"} err="failed to get container status \"160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951\": rpc error: code = NotFound desc = could not find container \"160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951\": container with ID starting with 160d86c4168df862c1ab2f3cf387377618beed7c3d7bcb59b0bd2175976dc951 not found: ID does not exist" Nov 02 03:05:36.819764 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.819736 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9"] Nov 02 03:05:36.825167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:36.825138 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9zv9"] Nov 02 03:05:38.113928 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:05:38.113897 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a941f7-6862-4602-a6dd-8516484742b8" path="/var/lib/kubelet/pods/93a941f7-6862-4602-a6dd-8516484742b8/volumes" Nov 02 03:06:03.447466 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:06:03.447425 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:06:33.444491 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:06:33.444451 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:07:02.782274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782199 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd"] Nov 02 03:07:02.782693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782535 4868 cpu_manager.go:414] "containerMap: removing container" podUID="93a941f7-6862-4602-a6dd-8516484742b8" containerName="manager" Nov 02 03:07:02.782693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782548 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a941f7-6862-4602-a6dd-8516484742b8" containerName="manager" Nov 02 03:07:02.782693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782565 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2ea3ee7f-2891-4662-9735-7ee636f1eae2" containerName="maas-api" Nov 02 03:07:02.782693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782571 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea3ee7f-2891-4662-9735-7ee636f1eae2" containerName="maas-api" Nov 02 03:07:02.782693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782634 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea3ee7f-2891-4662-9735-7ee636f1eae2" containerName="maas-api" Nov 02 03:07:02.782693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.782646 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a941f7-6862-4602-a6dd-8516484742b8" containerName="manager" Nov 02 03:07:02.785492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.785473 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.837394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.837364 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tk8sg\"" Nov 02 03:07:02.888935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.888905 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd"] Nov 02 03:07:02.897402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897378 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897423 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897552 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897522 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897552 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897547 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897576 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897671 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897760 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897808 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.897861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.897836 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grxl\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.998962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.998934 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.998962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.998966 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.998982 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grxl\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999005 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999027 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999066 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999089 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999130 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999164 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999397 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999538 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999587 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999655 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:02.999929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:02.999905 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:03.001319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:03.001300 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:03.001484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:03.001468 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:03.055390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:03.055320 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:03.055474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:03.055399 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grxl\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-xfdrd\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:03.097433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:03.097410 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:03.444043 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:07:03.443992 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:07:03.519166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:03.519143 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd"] Nov 02 03:07:03.521068 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:07:03.521041 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda827d914_fcc0_4f38_b440_b633f0794700.slice/crio-30ff594fbac61db1cffb76150378a8043774a48bf9dbf258535aa6ec426149f1 WatchSource:0}: Error finding container 30ff594fbac61db1cffb76150378a8043774a48bf9dbf258535aa6ec426149f1: Status 404 returned error can't find the container with id 30ff594fbac61db1cffb76150378a8043774a48bf9dbf258535aa6ec426149f1 Nov 02 03:07:04.020410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:04.020376 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" event={"ID":"a827d914-fcc0-4f38-b440-b633f0794700","Type":"ContainerStarted","Data":"30ff594fbac61db1cffb76150378a8043774a48bf9dbf258535aa6ec426149f1"} Nov 02 03:07:07.416308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:07.416278 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:07.416639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:07.416343 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:07.416639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:07.416400 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:08.060242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:08.060203 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" event={"ID":"a827d914-fcc0-4f38-b440-b633f0794700","Type":"ContainerStarted","Data":"4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d"} Nov 02 03:07:08.097906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:08.097879 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:08.099485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:08.099461 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:08.099617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:08.099507 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:08.267711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:08.267653 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podStartSLOduration=2.374976036 podStartE2EDuration="6.267639352s" totalImagesPullingTime="3.892663316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:08.225842458 +0000 UTC m=+1170.221543402" watchObservedRunningTime="2026-11-02 03:07:08.267639352 +0000 UTC m=+1170.263340294" Nov 02 03:07:09.097880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:09.097843 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:09.098295 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:09.097908 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:10.098246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:10.098213 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:10.098655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:10.098262 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:11.097853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:11.097819 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:11.098023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:11.097885 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:12.098767 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:12.098736 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:12.099147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:12.098802 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:13.098472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:13.098437 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:13.098654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:13.098489 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:14.097940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:14.097906 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 03:07:14.098435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:14.097968 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 03:07:15.589170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:15.589128 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:15.589661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:15.589520 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:15.590508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:15.590486 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:07:17.903542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:17.903504 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8"] Nov 02 03:07:17.907126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:17.907109 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.007747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.007716 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8"] Nov 02 03:07:18.048142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048109 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048276 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048151 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048276 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048177 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048270 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048313 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048331 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048438 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048456 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4wt\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.048489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.048487 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149498 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149529 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4wt\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149571 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149615 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149660 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149697 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149737 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149776 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.149883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149798 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.150084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.149988 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.150147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.150122 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.150199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.150145 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.150394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.150328 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.150394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.150393 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.151914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.151895 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.152157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.152140 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.205891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.205838 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.205981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.205897 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4wt\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt\") pod \"maas-default-gateway-openshift-default-58847f9fc5-6j5w8\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.219788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.219766 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:18.639314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.639290 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8"] Nov 02 03:07:18.640857 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:07:18.640833 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6db303b_b46b_4ad6_a2ce_d5e8d1a6ad29.slice/crio-cb3db43818c93fde2e40d1713eff45bd1042fa143d9e964d3af507305c847529 WatchSource:0}: Error finding container cb3db43818c93fde2e40d1713eff45bd1042fa143d9e964d3af507305c847529: Status 404 returned error can't find the container with id cb3db43818c93fde2e40d1713eff45bd1042fa143d9e964d3af507305c847529 Nov 02 03:07:18.642906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.642878 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:18.643002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.642938 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:18.643002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:18.642965 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:19.116760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:19.116729 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" event={"ID":"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29","Type":"ContainerStarted","Data":"3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97"} Nov 02 03:07:19.117158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:19.116788 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" event={"ID":"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29","Type":"ContainerStarted","Data":"cb3db43818c93fde2e40d1713eff45bd1042fa143d9e964d3af507305c847529"} Nov 02 03:07:19.220022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:19.219990 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:19.221412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:19.221388 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:19.221546 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:19.221433 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:19.233696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:19.233628 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podStartSLOduration=2.233608406 podStartE2EDuration="2.233608406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:19.231112055 +0000 UTC m=+1181.226812999" watchObservedRunningTime="2026-11-02 03:07:19.233608406 +0000 UTC m=+1181.229309351" Nov 02 03:07:20.221065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:20.221031 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:20.221469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:20.221091 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:21.222241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:21.222208 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:21.222654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:21.222272 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:22.220069 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:22.220029 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:22.220212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:22.220089 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:23.221031 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:23.220998 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:23.221482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:23.221054 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:24.220899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:24.220868 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:24.221075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:24.220927 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:25.220340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:25.220305 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 02 03:07:25.220549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:25.220380 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 02 03:07:26.832234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:26.832198 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:26.832738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:26.832669 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:26.833347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:26.833328 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:07:33.445409 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:07:33.445371 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:07:38.219937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:38.219908 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:07:38.222775 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:38.222754 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:07:43.703983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.703941 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh"] Nov 02 03:07:43.707585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.707558 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.771793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.771761 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.771953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.771799 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.771953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.771816 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.771953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.771912 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.772070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.772005 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.772070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.772042 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.772140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.772069 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6lr\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.772140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.772101 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.772140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.772118 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.819127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.819092 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh"] Nov 02 03:07:43.850781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.850748 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d1107301-openshift-default-dockercfg-drrll\"" Nov 02 03:07:43.872532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872503 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872539 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872557 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872772 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872849 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872891 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872915 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6lr\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.872954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872938 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.873113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872990 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.873113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.872946 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.873113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.873052 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.873294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.873274 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.873294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.873285 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.873886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.873864 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.875368 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.875335 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.875841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.875825 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.929790 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.929762 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:43.929916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:43.929802 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6lr\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr\") pod \"e2e-ait-d1107301-openshift-default-7c647db76c-2szsh\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:44.030519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:44.030438 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:44.481017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:44.480985 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh"] Nov 02 03:07:44.520343 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:07:44.520313 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d47e9_1a74_44e3_bda3_7d6686d3832a.slice/crio-32210dcb88ca7f52e0bd2312e6f3d22758504ef3add611b7125f14e0c8f88050 WatchSource:0}: Error finding container 32210dcb88ca7f52e0bd2312e6f3d22758504ef3add611b7125f14e0c8f88050: Status 404 returned error can't find the container with id 32210dcb88ca7f52e0bd2312e6f3d22758504ef3add611b7125f14e0c8f88050 Nov 02 03:07:44.522442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:44.522413 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:44.522531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:44.522474 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:44.522531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:44.522502 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:07:45.257747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:45.257699 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" event={"ID":"998d47e9-1a74-44e3-bda3-7d6686d3832a","Type":"ContainerStarted","Data":"01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320"} Nov 02 03:07:45.258135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:45.257766 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" event={"ID":"998d47e9-1a74-44e3-bda3-7d6686d3832a","Type":"ContainerStarted","Data":"32210dcb88ca7f52e0bd2312e6f3d22758504ef3add611b7125f14e0c8f88050"} Nov 02 03:07:45.377152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:45.377061 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podStartSLOduration=2.37704121 podStartE2EDuration="2.37704121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:07:45.376808457 +0000 UTC m=+1207.372509404" watchObservedRunningTime="2026-11-02 03:07:45.37704121 +0000 UTC m=+1207.372742155" Nov 02 03:07:46.031549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:46.031512 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:46.036270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:46.036244 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:46.264611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:46.263513 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:46.264611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:46.264536 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:07:58.862797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:58.862041 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr"] Nov 02 03:07:58.866057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:58.866032 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:58.929517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:58.929495 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-935e4abf-serving-cert\"" Nov 02 03:07:58.970288 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:58.970263 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pz7wg\"" Nov 02 03:07:58.970463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:58.970444 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 03:07:58.971962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:58.971942 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr"] Nov 02 03:07:59.021343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.021305 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.021540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.021440 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.021540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.021500 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtq6\" (UniqueName: \"kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.021540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.021521 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.122646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.122560 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.122646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.122607 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtq6\" (UniqueName: \"kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.122646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.122626 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.122926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.122689 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.122926 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:07:59.122788 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-serving-cert: secret "maas-api-e2e-migrate-935e4abf-serving-cert" not found Nov 02 03:07:59.122926 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:07:59.122848 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls podName:fb1cfc01-5b14-4f27-9199-6e3e60c5c847 nodeName:}" failed. No retries permitted until 2026-11-02 03:07:59.622833541 +0000 UTC m=+1221.618534462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls") pod "maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" (UID: "fb1cfc01-5b14-4f27-9199-6e3e60c5c847") : secret "maas-api-e2e-migrate-935e4abf-serving-cert" not found Nov 02 03:07:59.123299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.123275 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.125551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.125522 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.182863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.182840 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtq6\" (UniqueName: \"kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.631008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.630976 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.633329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.633306 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\") pod \"maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:07:59.779231 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:07:59.779198 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:08:00.145820 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:00.145797 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr"] Nov 02 03:08:00.148412 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:08:00.148387 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1cfc01_5b14_4f27_9199_6e3e60c5c847.slice/crio-b54a0bac8f7dc3ae3385ce6aa3aefb6925f8835485e2352a5075b98f2e2c0859 WatchSource:0}: Error finding container b54a0bac8f7dc3ae3385ce6aa3aefb6925f8835485e2352a5075b98f2e2c0859: Status 404 returned error can't find the container with id b54a0bac8f7dc3ae3385ce6aa3aefb6925f8835485e2352a5075b98f2e2c0859 Nov 02 03:08:00.350134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:00.350102 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" event={"ID":"fb1cfc01-5b14-4f27-9199-6e3e60c5c847","Type":"ContainerStarted","Data":"b54a0bac8f7dc3ae3385ce6aa3aefb6925f8835485e2352a5075b98f2e2c0859"} Nov 02 03:08:01.364551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:01.362390 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" event={"ID":"fb1cfc01-5b14-4f27-9199-6e3e60c5c847","Type":"ContainerStarted","Data":"529b51c41673a22eeaa3678bea1d2426896e2be9585123c946db827015126be7"} Nov 02 03:08:01.364551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:01.362537 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:08:01.476294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:01.476241 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" podStartSLOduration=2.577218306 podStartE2EDuration="3.476227539s" totalImagesPullingTime="899.009233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:01.474941261 +0000 UTC m=+1223.470642205" watchObservedRunningTime="2026-11-02 03:08:01.476227539 +0000 UTC m=+1223.471928526" Nov 02 03:08:03.445094 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:08:03.445058 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:08:07.373021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:07.372987 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:08:13.421248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:13.421205 4868 generic.go:356] "Generic (PLEG): container finished" podID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerID="01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320" exitCode=0 Nov 02 03:08:13.421682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:13.421272 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" event={"ID":"998d47e9-1a74-44e3-bda3-7d6686d3832a","Type":"ContainerDied","Data":"01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320"} Nov 02 03:08:13.421682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:13.421613 4868 scope.go:122] "RemoveContainer" containerID="01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320" Nov 02 03:08:13.422766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:13.422736 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:08:13.422860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:13.422804 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:08:13.422860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:13.422856 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:08:14.415431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.415401 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t"] Nov 02 03:08:14.418596 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.418576 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.428550 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.428526 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" event={"ID":"998d47e9-1a74-44e3-bda3-7d6686d3832a","Type":"ContainerStarted","Data":"c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20"} Nov 02 03:08:14.470402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.470325 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bc460382-serving-cert\"" Nov 02 03:08:14.521299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.521269 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t"] Nov 02 03:08:14.578150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.578116 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.578330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.578195 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.578619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.578594 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2bb\" (UniqueName: \"kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.578684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.578642 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.679894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.679796 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.680058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.679899 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2bb\" (UniqueName: \"kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.680058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.679933 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.680058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.679979 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.680465 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:08:14.680420 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-serving-cert: secret "maas-api-e2e-derive-bc460382-serving-cert" not found Nov 02 03:08:14.680621 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:08:14.680510 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls podName:809d2f79-714e-40b8-8d35-637494bd467a nodeName:}" failed. No retries permitted until 2026-11-02 03:08:15.180491637 +0000 UTC m=+1237.176192572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls") pod "maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" (UID: "809d2f79-714e-40b8-8d35-637494bd467a") : secret "maas-api-e2e-derive-bc460382-serving-cert" not found Nov 02 03:08:14.680814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.680785 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.683004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.682980 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:14.757692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:14.757666 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2bb\" (UniqueName: \"kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:15.030892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.030809 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:08:15.032049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.032018 4868 patch_prober.go:29] interesting pod/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 02 03:08:15.032287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.032259 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 02 03:08:15.184456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.184413 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:15.186900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.186871 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\") pod \"maas-api-e2e-derive-bc460382-7d857c487f-7lf4t\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:15.349599 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.349495 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:15.726384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.726347 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t"] Nov 02 03:08:15.728934 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:08:15.728902 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809d2f79_714e_40b8_8d35_637494bd467a.slice/crio-25e4a16e5eca935e60b39bde071065f1b2306603b38a84ddc313379399b5fe84 WatchSource:0}: Error finding container 25e4a16e5eca935e60b39bde071065f1b2306603b38a84ddc313379399b5fe84: Status 404 returned error can't find the container with id 25e4a16e5eca935e60b39bde071065f1b2306603b38a84ddc313379399b5fe84 Nov 02 03:08:15.730433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:15.730415 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:08:16.031036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:16.030960 4868 patch_prober.go:29] interesting pod/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 02 03:08:16.031186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:16.031020 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 02 03:08:16.264285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:16.264255 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:08:16.468951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:16.468908 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" event={"ID":"809d2f79-714e-40b8-8d35-637494bd467a","Type":"ContainerStarted","Data":"25e4a16e5eca935e60b39bde071065f1b2306603b38a84ddc313379399b5fe84"} Nov 02 03:08:17.031317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:17.031239 4868 patch_prober.go:29] interesting pod/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 02 03:08:17.031317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:17.031291 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 02 03:08:17.475690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:17.475639 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" event={"ID":"809d2f79-714e-40b8-8d35-637494bd467a","Type":"ContainerStarted","Data":"ed3c6ee0e6652623dbe3c4191ae03b24ef84f259bb367683c8bc772d74739815"} Nov 02 03:08:17.475867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:17.475797 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:17.591672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:17.591606 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" podStartSLOduration=2.627487591 podStartE2EDuration="3.591585957s" totalImagesPullingTime="964.098366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:17.588604939 +0000 UTC m=+1239.584305882" watchObservedRunningTime="2026-11-02 03:08:17.591585957 +0000 UTC m=+1239.587286901" Nov 02 03:08:18.031788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:18.031748 4868 patch_prober.go:29] interesting pod/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 02 03:08:18.032416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:18.032380 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 02 03:08:19.031736 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:19.031698 4868 patch_prober.go:29] interesting pod/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 02 03:08:19.032004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:19.031974 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 02 03:08:20.035340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:20.035303 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:08:20.036581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:20.036555 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:08:23.485807 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:23.485776 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:08:33.445582 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:08:33.445538 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:08:55.438022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.437987 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr"] Nov 02 03:08:55.438605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.438200 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" podUID="fb1cfc01-5b14-4f27-9199-6e3e60c5c847" containerName="maas-api" containerID="cri-o://529b51c41673a22eeaa3678bea1d2426896e2be9585123c946db827015126be7" gracePeriod=30 Nov 02 03:08:55.664426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.664166 4868 generic.go:356] "Generic (PLEG): container finished" podID="fb1cfc01-5b14-4f27-9199-6e3e60c5c847" containerID="529b51c41673a22eeaa3678bea1d2426896e2be9585123c946db827015126be7" exitCode=0 Nov 02 03:08:55.664426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.664317 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" event={"ID":"fb1cfc01-5b14-4f27-9199-6e3e60c5c847","Type":"ContainerDied","Data":"529b51c41673a22eeaa3678bea1d2426896e2be9585123c946db827015126be7"} Nov 02 03:08:55.699537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.699468 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:08:55.854524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.854490 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\") pod \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " Nov 02 03:08:55.854731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.854540 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca\") pod \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " Nov 02 03:08:55.854731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.854562 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs\") pod \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " Nov 02 03:08:55.854731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.854631 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6\" (UniqueName: \"kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6\") pod \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\" (UID: \"fb1cfc01-5b14-4f27-9199-6e3e60c5c847\") " Nov 02 03:08:55.854948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.854920 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca" pod "fb1cfc01-5b14-4f27-9199-6e3e60c5c847" (UID: "fb1cfc01-5b14-4f27-9199-6e3e60c5c847"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:08:55.856705 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.856677 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls" pod "fb1cfc01-5b14-4f27-9199-6e3e60c5c847" (UID: "fb1cfc01-5b14-4f27-9199-6e3e60c5c847"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:08:55.856805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.856746 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6" pod "fb1cfc01-5b14-4f27-9199-6e3e60c5c847" (UID: "fb1cfc01-5b14-4f27-9199-6e3e60c5c847"). InnerVolumeSpecName "kube-api-access-2vtq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:08:55.856805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.856769 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs" pod "fb1cfc01-5b14-4f27-9199-6e3e60c5c847" (UID: "fb1cfc01-5b14-4f27-9199-6e3e60c5c847"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:08:55.955973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.955910 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtq6\" (UniqueName: \"kubernetes.io/projected/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-kube-api-access-2vtq6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:08:55.955973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.955937 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:08:55.955973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.955948 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:08:55.955973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:55.955956 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb1cfc01-5b14-4f27-9199-6e3e60c5c847-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:08:56.671082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.671049 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" event={"ID":"fb1cfc01-5b14-4f27-9199-6e3e60c5c847","Type":"ContainerDied","Data":"b54a0bac8f7dc3ae3385ce6aa3aefb6925f8835485e2352a5075b98f2e2c0859"} Nov 02 03:08:56.671082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.671077 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr" Nov 02 03:08:56.671631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.671120 4868 scope.go:122] "RemoveContainer" containerID="529b51c41673a22eeaa3678bea1d2426896e2be9585123c946db827015126be7" Nov 02 03:08:56.686065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.686045 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh"] Nov 02 03:08:56.686237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.686219 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" containerID="cri-o://c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20" gracePeriod=30 Nov 02 03:08:56.947875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.947797 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr"] Nov 02 03:08:56.951554 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:56.951530 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-935e4abf-6f79cbbbdb-57fwr"] Nov 02 03:08:57.898452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:57.898415 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs"] Nov 02 03:08:57.898938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:57.898808 4868 cpu_manager.go:414] "containerMap: removing container" podUID="fb1cfc01-5b14-4f27-9199-6e3e60c5c847" containerName="maas-api" Nov 02 03:08:57.898938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:57.898821 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1cfc01-5b14-4f27-9199-6e3e60c5c847" containerName="maas-api" Nov 02 03:08:57.898938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:57.898886 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1cfc01-5b14-4f27-9199-6e3e60c5c847" containerName="maas-api" Nov 02 03:08:57.903216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:57.903197 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:57.976457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:57.976426 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs"] Nov 02 03:08:58.007553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.007525 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ca6546-openshift-default-dockercfg-c2jsr\"" Nov 02 03:08:58.076131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076082 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076128 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076148 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076266 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076325 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076405 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076437 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076525 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.076666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.076547 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p222\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.113638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.113609 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1cfc01-5b14-4f27-9199-6e3e60c5c847" path="/var/lib/kubelet/pods/fb1cfc01-5b14-4f27-9199-6e3e60c5c847/volumes" Nov 02 03:08:58.177955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.177886 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.177955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.177924 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.177955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.177956 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178072 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178103 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178156 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178186 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p222\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178260 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178285 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178335 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178430 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178610 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178709 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.178838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.178812 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.180565 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.180542 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.180804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.180780 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.234512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.234487 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.234631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.234557 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p222\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.515843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.515759 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:08:58.986635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.986613 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs"] Nov 02 03:08:58.988442 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:08:58.988412 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c7540a_eda8_4b7b_8ac3_a46bbace2b72.slice/crio-659dad4160f1f13881d60fa6a9be67e36d2e575bbfefaa64a75129a8c8b9b6c3 WatchSource:0}: Error finding container 659dad4160f1f13881d60fa6a9be67e36d2e575bbfefaa64a75129a8c8b9b6c3: Status 404 returned error can't find the container with id 659dad4160f1f13881d60fa6a9be67e36d2e575bbfefaa64a75129a8c8b9b6c3 Nov 02 03:08:58.990475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.990442 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:08:58.990541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.990509 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:08:58.990578 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:58.990542 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:08:59.689855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:59.689817 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" event={"ID":"00c7540a-eda8-4b7b-8ac3-a46bbace2b72","Type":"ContainerStarted","Data":"126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf"} Nov 02 03:08:59.690047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:08:59.689889 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" event={"ID":"00c7540a-eda8-4b7b-8ac3-a46bbace2b72","Type":"ContainerStarted","Data":"659dad4160f1f13881d60fa6a9be67e36d2e575bbfefaa64a75129a8c8b9b6c3"} Nov 02 03:09:00.515940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:00.515903 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:00.521240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:00.521215 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:00.686602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:00.686552 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podStartSLOduration=3.686538925 podStartE2EDuration="3.686538925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:08:59.868683322 +0000 UTC m=+1281.864384266" watchObservedRunningTime="2026-11-02 03:09:00.686538925 +0000 UTC m=+1282.682239867" Nov 02 03:09:00.695088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:00.695058 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:00.696118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:00.696095 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:01.942558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:01.942532 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:09:02.014223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014134 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014176 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014213 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014237 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014253 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014275 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014324 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014364 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014468 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:09:02.014542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014507 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:09:02.014864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014554 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs\") pod \"998d47e9-1a74-44e3-bda3-7d6686d3832a\" (UID: \"998d47e9-1a74-44e3-bda3-7d6686d3832a\") " Nov 02 03:09:02.014864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014585 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:09:02.014864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014658 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:09:02.014864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014798 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:09:02.015012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014910 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.015012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014928 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.015012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014941 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.015012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014956 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998d47e9-1a74-44e3-bda3-7d6686d3832a-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.015012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.014968 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.016730 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.016700 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "kube-api-access-rm6lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:09:02.016874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.016744 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:09:02.016874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.016759 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:09:02.017379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.017327 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo" pod "998d47e9-1a74-44e3-bda3-7d6686d3832a" (UID: "998d47e9-1a74-44e3-bda3-7d6686d3832a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:09:02.115926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.115892 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.115926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.115926 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.116167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.115941 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.116167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.115955 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm6lr\" (UniqueName: \"kubernetes.io/projected/998d47e9-1a74-44e3-bda3-7d6686d3832a-kube-api-access-rm6lr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:02.705051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.705021 4868 generic.go:356] "Generic (PLEG): container finished" podID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerID="c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20" exitCode=0 Nov 02 03:09:02.705217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.705074 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" Nov 02 03:09:02.705217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.705099 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" event={"ID":"998d47e9-1a74-44e3-bda3-7d6686d3832a","Type":"ContainerDied","Data":"c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20"} Nov 02 03:09:02.705217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.705152 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh" event={"ID":"998d47e9-1a74-44e3-bda3-7d6686d3832a","Type":"ContainerDied","Data":"32210dcb88ca7f52e0bd2312e6f3d22758504ef3add611b7125f14e0c8f88050"} Nov 02 03:09:02.705217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.705175 4868 scope.go:122] "RemoveContainer" containerID="c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20" Nov 02 03:09:02.713975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.713955 4868 scope.go:122] "RemoveContainer" containerID="01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320" Nov 02 03:09:02.721667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.721650 4868 scope.go:122] "RemoveContainer" containerID="c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20" Nov 02 03:09:02.721927 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:09:02.721910 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20\": container with ID starting with c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20 not found: ID does not exist" containerID="c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20" Nov 02 03:09:02.721974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.721940 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20"} err="failed to get container status \"c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20\": rpc error: code = NotFound desc = could not find container \"c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20\": container with ID starting with c9a94cb6d9a9039f94cde62fcb4b5436d6158e0faaf40796631a42e3ce7c7c20 not found: ID does not exist" Nov 02 03:09:02.721974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.721957 4868 scope.go:122] "RemoveContainer" containerID="01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320" Nov 02 03:09:02.722187 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:09:02.722170 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320\": container with ID starting with 01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320 not found: ID does not exist" containerID="01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320" Nov 02 03:09:02.722230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.722196 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320"} err="failed to get container status \"01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320\": rpc error: code = NotFound desc = could not find container \"01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320\": container with ID starting with 01cafe053f9ac2cf868e92dc1d3c7e074f7b32484f9199f1cd8b5b9d340f2320 not found: ID does not exist" Nov 02 03:09:02.932631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.932593 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh"] Nov 02 03:09:02.981824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:02.981791 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d1107301-openshift-default-7c647db76c-2szsh"] Nov 02 03:09:03.445383 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:09:03.445333 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:09:04.113686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:04.113658 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" path="/var/lib/kubelet/pods/998d47e9-1a74-44e3-bda3-7d6686d3832a/volumes" Nov 02 03:09:24.194563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.194491 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c"] Nov 02 03:09:24.194916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.194838 4868 cpu_manager.go:414] "containerMap: removing container" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" Nov 02 03:09:24.194916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.194850 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" Nov 02 03:09:24.194916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.194877 4868 cpu_manager.go:414] "containerMap: removing container" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" Nov 02 03:09:24.194916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.194885 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" Nov 02 03:09:24.195041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.194946 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" Nov 02 03:09:24.197962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.197945 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.252292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.252271 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-858236-serving-cert\"" Nov 02 03:09:24.303424 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.303398 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c"] Nov 02 03:09:24.326836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.326809 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.326943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.326860 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.326943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.326880 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgjx\" (UniqueName: \"kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.327017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.327002 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.428403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.428375 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.428564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.428454 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.428564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.428485 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgjx\" (UniqueName: \"kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.428564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.428522 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.428733 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:09:24.428588 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-serving-cert: secret "maas-api-e2e-worker-w3-858236-serving-cert" not found Nov 02 03:09:24.428733 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:09:24.428665 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls podName:b5811b16-e35e-450f-bf28-543513ce47da nodeName:}" failed. No retries permitted until 2026-11-02 03:09:24.928645064 +0000 UTC m=+1306.924346004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls") pod "maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" (UID: "b5811b16-e35e-450f-bf28-543513ce47da") : secret "maas-api-e2e-worker-w3-858236-serving-cert" not found Nov 02 03:09:24.429160 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.429136 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.430767 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.430746 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.485377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.485294 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgjx\" (UniqueName: \"kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.931741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.931705 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:24.934195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:24.934174 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:25.110081 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:25.110053 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:25.479532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:25.479455 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c"] Nov 02 03:09:25.481665 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:09:25.481638 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5811b16_e35e_450f_bf28_543513ce47da.slice/crio-595cc1646da51a3f97c85eb5c7e2d70bd6d5821d1482c7cceb63b0cf12d05a22 WatchSource:0}: Error finding container 595cc1646da51a3f97c85eb5c7e2d70bd6d5821d1482c7cceb63b0cf12d05a22: Status 404 returned error can't find the container with id 595cc1646da51a3f97c85eb5c7e2d70bd6d5821d1482c7cceb63b0cf12d05a22 Nov 02 03:09:25.840532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:25.840450 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" event={"ID":"b5811b16-e35e-450f-bf28-543513ce47da","Type":"ContainerStarted","Data":"595cc1646da51a3f97c85eb5c7e2d70bd6d5821d1482c7cceb63b0cf12d05a22"} Nov 02 03:09:26.846579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:26.846484 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" event={"ID":"b5811b16-e35e-450f-bf28-543513ce47da","Type":"ContainerStarted","Data":"1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b"} Nov 02 03:09:26.847011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:26.846609 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:26.960001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:26.959944 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" podStartSLOduration=2.003935507 podStartE2EDuration="2.959927366s" totalImagesPullingTime="955.991859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:26.958920891 +0000 UTC m=+1308.954621833" watchObservedRunningTime="2026-11-02 03:09:26.959927366 +0000 UTC m=+1308.955628313" Nov 02 03:09:28.594512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.594482 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t"] Nov 02 03:09:28.594895 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.594682 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" podUID="809d2f79-714e-40b8-8d35-637494bd467a" containerName="maas-api" containerID="cri-o://ed3c6ee0e6652623dbe3c4191ae03b24ef84f259bb367683c8bc772d74739815" gracePeriod=30 Nov 02 03:09:28.859842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.859816 4868 generic.go:356] "Generic (PLEG): container finished" podID="809d2f79-714e-40b8-8d35-637494bd467a" containerID="ed3c6ee0e6652623dbe3c4191ae03b24ef84f259bb367683c8bc772d74739815" exitCode=0 Nov 02 03:09:28.859961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.859856 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" event={"ID":"809d2f79-714e-40b8-8d35-637494bd467a","Type":"ContainerDied","Data":"ed3c6ee0e6652623dbe3c4191ae03b24ef84f259bb367683c8bc772d74739815"} Nov 02 03:09:28.859961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.859898 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" event={"ID":"809d2f79-714e-40b8-8d35-637494bd467a","Type":"ContainerDied","Data":"25e4a16e5eca935e60b39bde071065f1b2306603b38a84ddc313379399b5fe84"} Nov 02 03:09:28.859961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.859929 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e4a16e5eca935e60b39bde071065f1b2306603b38a84ddc313379399b5fe84" Nov 02 03:09:28.864545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.864525 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:09:28.966272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.966247 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca\") pod \"809d2f79-714e-40b8-8d35-637494bd467a\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " Nov 02 03:09:28.966272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.966277 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\") pod \"809d2f79-714e-40b8-8d35-637494bd467a\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " Nov 02 03:09:28.966478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.966402 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs\") pod \"809d2f79-714e-40b8-8d35-637494bd467a\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " Nov 02 03:09:28.966478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.966465 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb\" (UniqueName: \"kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb\") pod \"809d2f79-714e-40b8-8d35-637494bd467a\" (UID: \"809d2f79-714e-40b8-8d35-637494bd467a\") " Nov 02 03:09:28.966666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.966644 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca" pod "809d2f79-714e-40b8-8d35-637494bd467a" (UID: "809d2f79-714e-40b8-8d35-637494bd467a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:09:28.966796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.966781 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/809d2f79-714e-40b8-8d35-637494bd467a-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:28.968283 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.968262 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls" pod "809d2f79-714e-40b8-8d35-637494bd467a" (UID: "809d2f79-714e-40b8-8d35-637494bd467a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:09:28.968373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.968309 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs" pod "809d2f79-714e-40b8-8d35-637494bd467a" (UID: "809d2f79-714e-40b8-8d35-637494bd467a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:09:28.968443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:28.968427 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb" pod "809d2f79-714e-40b8-8d35-637494bd467a" (UID: "809d2f79-714e-40b8-8d35-637494bd467a"). InnerVolumeSpecName "kube-api-access-cl2bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:09:29.067254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:29.067220 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:29.067254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:29.067248 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl2bb\" (UniqueName: \"kubernetes.io/projected/809d2f79-714e-40b8-8d35-637494bd467a-kube-api-access-cl2bb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:29.067254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:29.067257 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/809d2f79-714e-40b8-8d35-637494bd467a-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:09:29.864870 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:29.864840 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t" Nov 02 03:09:30.028219 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:30.028182 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t"] Nov 02 03:09:30.077617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:30.077586 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc460382-7d857c487f-7lf4t"] Nov 02 03:09:30.113344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:30.113320 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809d2f79-714e-40b8-8d35-637494bd467a" path="/var/lib/kubelet/pods/809d2f79-714e-40b8-8d35-637494bd467a/volumes" Nov 02 03:09:32.071295 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.071259 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2"] Nov 02 03:09:32.071787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.071639 4868 cpu_manager.go:414] "containerMap: removing container" podUID="809d2f79-714e-40b8-8d35-637494bd467a" containerName="maas-api" Nov 02 03:09:32.071787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.071652 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809d2f79-714e-40b8-8d35-637494bd467a" containerName="maas-api" Nov 02 03:09:32.071787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.071728 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="998d47e9-1a74-44e3-bda3-7d6686d3832a" containerName="istio-proxy" Nov 02 03:09:32.071787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.071735 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="809d2f79-714e-40b8-8d35-637494bd467a" containerName="maas-api" Nov 02 03:09:32.075633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.075613 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.127266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.127231 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5ac93f-openshift-default-dockercfg-cjnld\"" Nov 02 03:09:32.179477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.179440 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2"] Nov 02 03:09:32.193508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193481 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193545 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193654 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193707 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193758 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193785 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193813 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2qg\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193845 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.193946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.193907 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.294843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.294804 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.295214 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.295159 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.295429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.295406 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.295590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.295574 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.295699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.295685 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.295869 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.295847 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.295995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.295968 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.296130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.296014 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2qg\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.296130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.296046 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.296548 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.296525 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.296711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.296686 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.297040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.297023 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.297284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.297145 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.297481 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.297459 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.299536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.299515 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.299814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.299793 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.356476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.356447 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.356658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.356636 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2qg\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.388313 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.388293 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:32.809068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.809029 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2"] Nov 02 03:09:32.811570 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:09:32.811401 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b3ebbb_20a1_442c_8ed7_7c930929d773.slice/crio-7ef05c9a4777c5ecc46f8ac58fd321bbb6bde1c530812c272320c3c64c53fabf WatchSource:0}: Error finding container 7ef05c9a4777c5ecc46f8ac58fd321bbb6bde1c530812c272320c3c64c53fabf: Status 404 returned error can't find the container with id 7ef05c9a4777c5ecc46f8ac58fd321bbb6bde1c530812c272320c3c64c53fabf Nov 02 03:09:32.814170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.814132 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:09:32.814252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.814202 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:09:32.814252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.814233 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:09:32.867051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.867017 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 03:09:32.883607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:32.883564 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" event={"ID":"83b3ebbb-20a1-442c-8ed7-7c930929d773","Type":"ContainerStarted","Data":"7ef05c9a4777c5ecc46f8ac58fd321bbb6bde1c530812c272320c3c64c53fabf"} Nov 02 03:09:33.467099 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:09:33.467063 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:09:33.892051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:33.892012 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" event={"ID":"83b3ebbb-20a1-442c-8ed7-7c930929d773","Type":"ContainerStarted","Data":"616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238"} Nov 02 03:09:34.113865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:34.113803 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podStartSLOduration=2.113783059 podStartE2EDuration="2.113783059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:09:34.06286753 +0000 UTC m=+1316.058568486" watchObservedRunningTime="2026-11-02 03:09:34.113783059 +0000 UTC m=+1316.109484006" Nov 02 03:09:34.388972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:34.388927 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:34.393893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:34.393868 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:34.898844 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:34.898813 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:34.899750 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:34.899726 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:09:41.933523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:41.933486 4868 generic.go:356] "Generic (PLEG): container finished" podID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerID="126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf" exitCode=0 Nov 02 03:09:41.934061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:41.933529 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" event={"ID":"00c7540a-eda8-4b7b-8ac3-a46bbace2b72","Type":"ContainerDied","Data":"126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf"} Nov 02 03:09:41.934061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:41.933939 4868 scope.go:122] "RemoveContainer" containerID="126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf" Nov 02 03:09:41.935131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:41.935102 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:09:41.935236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:41.935186 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:09:41.935298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:41.935233 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:09:42.974153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:42.974118 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" event={"ID":"00c7540a-eda8-4b7b-8ac3-a46bbace2b72","Type":"ContainerStarted","Data":"e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd"} Nov 02 03:09:43.516516 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:43.516488 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:43.517793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:43.517767 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 03:09:43.517943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:43.517833 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 03:09:44.516610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:44.516569 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 03:09:44.517000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:44.516631 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 03:09:45.516766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:45.516737 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 03:09:45.517150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:45.516786 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 03:09:45.694964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:45.694922 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:46.516521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:46.516480 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 03:09:46.516693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:46.516540 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 03:09:47.516743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:47.516708 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 02 03:09:47.517148 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:47.516782 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 02 03:09:48.520562 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:48.520531 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:09:48.521437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:09:48.521416 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:10:03.447637 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:10:03.447601 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:10:20.940537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:20.940489 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:10:20.941100 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:20.940571 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:10:23.171336 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.171298 4868 generic.go:356] "Generic (PLEG): container finished" podID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerID="616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238" exitCode=0 Nov 02 03:10:23.171796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.171387 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" event={"ID":"83b3ebbb-20a1-442c-8ed7-7c930929d773","Type":"ContainerDied","Data":"616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238"} Nov 02 03:10:23.171796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.171736 4868 scope.go:122] "RemoveContainer" containerID="616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238" Nov 02 03:10:23.172973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.172948 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:10:23.173028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.173015 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:10:23.173063 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.173046 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:10:23.209934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.209914 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns"] Nov 02 03:10:23.215117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.215095 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.274729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.274699 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"openshift-service-ca.crt\"" Nov 02 03:10:23.276971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.276943 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns"] Nov 02 03:10:23.277159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.277135 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"kube-root-ca.crt\"" Nov 02 03:10:23.307340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.307315 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"simulator-w3-858236-kserve-self-signed-certs\"" Nov 02 03:10:23.307490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.307342 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"default-dockercfg-qtpc8\"" Nov 02 03:10:23.314605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.314582 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.314716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.314651 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.314716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.314696 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9k8\" (UniqueName: \"kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.314848 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.314820 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.314949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.314884 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.315038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.315018 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416078 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416136 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416161 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9k8\" (UniqueName: \"kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416202 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416243 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416306 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416619 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416919 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.416943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.416937 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.418327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.418303 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.418802 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.418779 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.478006 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.477553 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9k8\" (UniqueName: \"kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8\") pod \"simulator-w3-858236-kserve-755f99b74-t7jns\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.526536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.526503 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:23.908942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:23.908915 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns"] Nov 02 03:10:23.911228 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:10:23.911195 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeef3ea2_666b_406e_90f2_a059512f1a90.slice/crio-3e0c7fa500151cba37d5e0297851d61109bb9778f9815e1d58fad0f96e659d23 WatchSource:0}: Error finding container 3e0c7fa500151cba37d5e0297851d61109bb9778f9815e1d58fad0f96e659d23: Status 404 returned error can't find the container with id 3e0c7fa500151cba37d5e0297851d61109bb9778f9815e1d58fad0f96e659d23 Nov 02 03:10:24.177914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:24.177821 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" event={"ID":"83b3ebbb-20a1-442c-8ed7-7c930929d773","Type":"ContainerStarted","Data":"4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4"} Nov 02 03:10:24.179055 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:24.179032 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" event={"ID":"eeef3ea2-666b-406e-90f2-a059512f1a90","Type":"ContainerStarted","Data":"3e0c7fa500151cba37d5e0297851d61109bb9778f9815e1d58fad0f96e659d23"} Nov 02 03:10:24.388774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:24.388741 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:10:24.389043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:24.389022 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 02 03:10:24.389096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:24.389068 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 02 03:10:25.389086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:25.389048 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 02 03:10:25.389495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:25.389113 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 02 03:10:26.388673 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:26.388634 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 02 03:10:26.388848 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:26.388689 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 02 03:10:27.194666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:27.194631 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" event={"ID":"eeef3ea2-666b-406e-90f2-a059512f1a90","Type":"ContainerStarted","Data":"62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6"} Nov 02 03:10:27.312342 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:27.312285 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podStartSLOduration=2.049956063 podStartE2EDuration="4.312268052s" totalImagesPullingTime="2.262311989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:10:27.310900856 +0000 UTC m=+1369.306601799" watchObservedRunningTime="2026-11-02 03:10:27.312268052 +0000 UTC m=+1369.307968995" Nov 02 03:10:27.389199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:27.389167 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 02 03:10:27.389375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:27.389215 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 02 03:10:27.527385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:27.527299 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:28.388701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:28.388668 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 02 03:10:28.389170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:28.388719 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 02 03:10:29.409659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:29.409180 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:10:29.410154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:29.409737 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:10:29.410642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:29.410619 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 03:10:33.445465 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:10:33.445433 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:10:33.527508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:33.527478 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:33.539772 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:33.539747 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:34.240588 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:34.240557 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:10:42.656640 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.656606 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x"] Nov 02 03:10:42.660014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.659996 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.707135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.707107 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.707135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.707136 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smm98\" (UniqueName: \"kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.707168 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.707387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.707347 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.707453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.707405 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.707493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.707468 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.763755 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.763725 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"openshift-service-ca.crt\"" Nov 02 03:10:42.763900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.763796 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"default-dockercfg-cm7t6\"" Nov 02 03:10:42.763900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.763809 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"kube-root-ca.crt\"" Nov 02 03:10:42.763900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.763813 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"premium-w2-ca6546-kserve-self-signed-certs\"" Nov 02 03:10:42.765438 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.765415 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x"] Nov 02 03:10:42.817140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.817105 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.818977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.818941 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smm98\" (UniqueName: \"kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.819211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.819182 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.819329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.819259 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.819329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.819296 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.819465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.819327 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.820079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.820056 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.820271 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.820095 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.820427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.820403 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.822319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.822295 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.823163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.823141 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.875815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.875791 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smm98\" (UniqueName: \"kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98\") pod \"premium-w2-ca6546-kserve-6844f9d785-qtk5x\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:42.971078 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:42.970984 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:43.361260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:43.361236 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x"] Nov 02 03:10:43.363308 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:10:43.363284 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e4f2d56_ebda_4544_9c44_063ebd385386.slice/crio-e17f12a7bd19b61e5328e1130d2682785aa0e2f462afa7f77e6a669c3517bd34 WatchSource:0}: Error finding container e17f12a7bd19b61e5328e1130d2682785aa0e2f462afa7f77e6a669c3517bd34: Status 404 returned error can't find the container with id e17f12a7bd19b61e5328e1130d2682785aa0e2f462afa7f77e6a669c3517bd34 Nov 02 03:10:44.281225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:44.281190 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" event={"ID":"1e4f2d56-ebda-4544-9c44-063ebd385386","Type":"ContainerStarted","Data":"c6092239ba80cfc2d6715b135311a22462c2e8999e30ae2ca4e733d2e926b1f2"} Nov 02 03:10:44.281225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:44.281231 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" event={"ID":"1e4f2d56-ebda-4544-9c44-063ebd385386","Type":"ContainerStarted","Data":"e17f12a7bd19b61e5328e1130d2682785aa0e2f462afa7f77e6a669c3517bd34"} Nov 02 03:10:44.397196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:44.397135 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podStartSLOduration=2.397120846 podStartE2EDuration="2.397120846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:10:44.395336543 +0000 UTC m=+1386.391037486" watchObservedRunningTime="2026-11-02 03:10:44.397120846 +0000 UTC m=+1386.392821790" Nov 02 03:10:44.972157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:44.972119 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:52.971929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:52.971849 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:52.983891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:52.983866 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:10:53.334119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:10:53.334047 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:11:03.444620 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:11:03.444584 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:11:16.600127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.600085 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg"] Nov 02 03:11:16.605343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.605319 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.625420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.625399 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.625531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.625456 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.625577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.625523 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxbd\" (UniqueName: \"kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.625636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.625580 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.625636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.625597 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.625768 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.625745 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.658921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.658848 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f6e39\"/\"openshift-service-ca.crt\"" Nov 02 03:11:16.708386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.708347 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f6e39\"/\"default-dockercfg-tb4qg\"" Nov 02 03:11:16.708559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.708543 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f6e39\"/\"kube-root-ca.crt\"" Nov 02 03:11:16.708662 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.708645 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f6e39\"/\"simulator-w0-1f6e39-kserve-self-signed-certs\"" Nov 02 03:11:16.709943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.709918 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg"] Nov 02 03:11:16.727018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.726995 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727035 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxbd\" (UniqueName: \"kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727098 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727231 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727131 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727231 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727191 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727238 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727448 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727507 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.727608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.727587 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.729404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.729382 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.729630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.729612 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.808839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.808796 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxbd\" (UniqueName: \"kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd\") pod \"simulator-w0-1f6e39-kserve-696c494c9c-xbwkg\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:16.916783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:16.916739 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:17.298683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:17.298655 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg"] Nov 02 03:11:17.300750 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:11:17.300719 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcd478ed_4edd_4b4f_a11e_8ce86f9321cd.slice/crio-f35e8adfb88d4221aee640a7b614636da3321e40ccedcdf1775731201f7736c9 WatchSource:0}: Error finding container f35e8adfb88d4221aee640a7b614636da3321e40ccedcdf1775731201f7736c9: Status 404 returned error can't find the container with id f35e8adfb88d4221aee640a7b614636da3321e40ccedcdf1775731201f7736c9 Nov 02 03:11:17.435246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:17.435217 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" event={"ID":"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd","Type":"ContainerStarted","Data":"500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335"} Nov 02 03:11:17.435450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:17.435255 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" event={"ID":"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd","Type":"ContainerStarted","Data":"f35e8adfb88d4221aee640a7b614636da3321e40ccedcdf1775731201f7736c9"} Nov 02 03:11:17.549394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:17.549288 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podStartSLOduration=1.549273199 podStartE2EDuration="1.549273199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:11:17.547945794 +0000 UTC m=+1419.543646739" watchObservedRunningTime="2026-11-02 03:11:17.549273199 +0000 UTC m=+1419.544974142" Nov 02 03:11:17.917188 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:17.917156 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:25.844486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.844451 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s"] Nov 02 03:11:25.848260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.848228 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.899983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.899954 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.900125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.900063 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.900170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.900156 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.900201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.900181 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlwk\" (UniqueName: \"kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.900238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.900209 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.900329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.900308 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:25.951607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.951571 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"premium-w3-858236-kserve-self-signed-certs\"" Nov 02 03:11:25.953080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:25.953057 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s"] Nov 02 03:11:26.002584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.002543 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.002892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.002634 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.002892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.002671 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlwk\" (UniqueName: \"kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.002892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.002709 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.002892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.002740 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.002892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.002794 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.003273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.003243 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.003528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.003511 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.003812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.003779 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.008796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.007821 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.009591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.009568 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.059286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.059261 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlwk\" (UniqueName: \"kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk\") pod \"premium-w3-858236-kserve-54dd8c6599-ld76s\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.161071 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.161041 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:26.536155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.536130 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s"] Nov 02 03:11:26.537621 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:11:26.537595 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod367ac7f7_72db_466f_a2b9_8a45e54f32d8.slice/crio-df04693ed4ae02470f4ed631a70f498cad0d8a277c8044d417565dfd2d0636be WatchSource:0}: Error finding container df04693ed4ae02470f4ed631a70f498cad0d8a277c8044d417565dfd2d0636be: Status 404 returned error can't find the container with id df04693ed4ae02470f4ed631a70f498cad0d8a277c8044d417565dfd2d0636be Nov 02 03:11:26.916976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.916942 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:26.929587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:26.929555 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:27.489291 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:27.489248 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" event={"ID":"367ac7f7-72db-466f-a2b9-8a45e54f32d8","Type":"ContainerStarted","Data":"00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a"} Nov 02 03:11:27.489495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:27.489300 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" event={"ID":"367ac7f7-72db-466f-a2b9-8a45e54f32d8","Type":"ContainerStarted","Data":"df04693ed4ae02470f4ed631a70f498cad0d8a277c8044d417565dfd2d0636be"} Nov 02 03:11:27.500556 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:27.500533 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:11:27.603949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:27.603903 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podStartSLOduration=2.603891249 podStartE2EDuration="2.603891249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:11:27.602037568 +0000 UTC m=+1429.597738513" watchObservedRunningTime="2026-11-02 03:11:27.603891249 +0000 UTC m=+1429.599592192" Nov 02 03:11:28.161173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:28.161145 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:33.446012 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:11:33.445968 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:11:36.161758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:36.161705 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:36.174757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:36.174734 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:11:36.544185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:11:36.544112 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:12:02.399417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.399384 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc"] Nov 02 03:12:02.403039 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.403022 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.454994 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.454970 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-858236-openshift-default-dockercfg-79wxp\"" Nov 02 03:12:02.506875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.506843 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc"] Nov 02 03:12:02.530838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.530807 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.530979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.530852 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.530993 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531062 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.531035 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.531060 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.531079 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.531098 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.531225 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.531272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.531249 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljwx\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.631719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.631682 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljwx\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.631874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.631749 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.631874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.631787 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.631974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.631953 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632006 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632033 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632052 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632079 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632140 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632201 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632305 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632336 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632427 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.632930 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.632901 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.634812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.634786 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.635044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.635025 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.688816 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.688759 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljwx\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.688944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.688862 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:02.715412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:02.715387 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:03.135984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.135958 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc"] Nov 02 03:12:03.137851 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:12:03.137822 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e7f592_6cca_4fff_8711_ea3f7f39a578.slice/crio-e64fc1f6a3f4b315047d392c90de92e4fff65bee12f07b3fa976bb96ad3e5875 WatchSource:0}: Error finding container e64fc1f6a3f4b315047d392c90de92e4fff65bee12f07b3fa976bb96ad3e5875: Status 404 returned error can't find the container with id e64fc1f6a3f4b315047d392c90de92e4fff65bee12f07b3fa976bb96ad3e5875 Nov 02 03:12:03.139997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.139967 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:12:03.140089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.140027 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:12:03.140089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.140054 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:12:03.447408 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:12:03.447294 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:12:03.662568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.662534 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" event={"ID":"03e7f592-6cca-4fff-8711-ea3f7f39a578","Type":"ContainerStarted","Data":"e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73"} Nov 02 03:12:03.662733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.662586 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" event={"ID":"03e7f592-6cca-4fff-8711-ea3f7f39a578","Type":"ContainerStarted","Data":"e64fc1f6a3f4b315047d392c90de92e4fff65bee12f07b3fa976bb96ad3e5875"} Nov 02 03:12:03.716322 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.716244 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:03.779405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:03.779282 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podStartSLOduration=1.779266249 podStartE2EDuration="1.779266249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:12:03.778665531 +0000 UTC m=+1465.774366474" watchObservedRunningTime="2026-11-02 03:12:03.779266249 +0000 UTC m=+1465.774967195" Nov 02 03:12:04.718754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:04.718713 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 03:12:04.719344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:04.719306 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 03:12:05.716097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:05.716064 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 03:12:05.716284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:05.716113 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 03:12:06.715901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:06.715867 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 03:12:06.716288 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:06.715920 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 03:12:07.715696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:07.715663 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 03:12:07.715951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:07.715718 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 03:12:08.716480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:08.716446 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 03:12:08.716851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:08.716498 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 03:12:10.409103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:10.409060 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:10.409908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:10.409878 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:10.410781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:10.410760 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:12:33.447365 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:12:33.447306 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:12:38.252935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:38.252906 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:12:38.256230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:12:38.256207 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:13:03.445213 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:13:03.445177 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:13:33.447720 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:13:33.447679 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:13:35.694290 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.694253 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps"] Nov 02 03:13:35.698296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.698274 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.729912 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.729879 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.729921 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.729963 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.729989 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp545\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.730007 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.730024 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.730040 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.730068 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.730248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.730104 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.752346 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.751674 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f6e39-openshift-default-dockercfg-qc8xh\"" Nov 02 03:13:35.812470 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.812431 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps"] Nov 02 03:13:35.832789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.832761 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.833380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.833345 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.834005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.833984 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.834509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.834493 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp545\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.834894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.834878 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.835376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.835341 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.835817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.835802 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.836422 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.836406 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.837298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.836138 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.837432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.835756 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.837537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.836325 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.837612 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.834392 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.837684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.837158 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.837777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.837246 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.838152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.838079 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.841226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.841173 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.890914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.890887 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:35.891187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:35.891165 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp545\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:36.015210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:36.015115 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:36.445871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:36.445843 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps"] Nov 02 03:13:36.447543 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:13:36.447516 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb9009a_ce66_424c_ab6f_26bd8e115b29.slice/crio-c1d0356ec0085f5a415af32bbd95517ca62b9019ec084fb4eb2efd6a0592c771 WatchSource:0}: Error finding container c1d0356ec0085f5a415af32bbd95517ca62b9019ec084fb4eb2efd6a0592c771: Status 404 returned error can't find the container with id c1d0356ec0085f5a415af32bbd95517ca62b9019ec084fb4eb2efd6a0592c771 Nov 02 03:13:36.449837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:36.449806 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:13:36.449888 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:36.449870 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:13:36.449923 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:36.449899 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:13:37.179283 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:37.179249 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" event={"ID":"4fb9009a-ce66-424c-ab6f-26bd8e115b29","Type":"ContainerStarted","Data":"64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d"} Nov 02 03:13:37.179757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:37.179736 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" event={"ID":"4fb9009a-ce66-424c-ab6f-26bd8e115b29","Type":"ContainerStarted","Data":"c1d0356ec0085f5a415af32bbd95517ca62b9019ec084fb4eb2efd6a0592c771"} Nov 02 03:13:37.393582 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:37.393520 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podStartSLOduration=2.393502707 podStartE2EDuration="2.393502707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:37.34290168 +0000 UTC m=+1559.338602624" watchObservedRunningTime="2026-11-02 03:13:37.393502707 +0000 UTC m=+1559.389203651" Nov 02 03:13:38.015715 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:38.015677 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:38.017076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:38.017055 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 02 03:13:38.017148 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:38.017101 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 02 03:13:39.016324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:39.016296 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 02 03:13:39.016706 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:39.016344 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 02 03:13:40.015840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:40.015807 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 02 03:13:40.016023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:40.015859 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 02 03:13:41.016451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:41.016413 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 02 03:13:41.016848 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:41.016468 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 02 03:13:42.016445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:42.016411 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 02 03:13:42.016655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:42.016472 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 02 03:13:43.179098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:43.179066 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:43.179957 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:43.179848 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:43.180811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:43.180791 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:13:50.871485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.871406 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk"] Nov 02 03:13:50.874924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.874906 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950222 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950257 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950287 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950431 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950550 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950471 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgk68\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950550 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950509 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950616 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950673 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950641 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:50.950718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:50.950673 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051698 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051737 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051759 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051787 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051803 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgk68\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051828 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051886 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051909 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.051963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.051940 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.052370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.052320 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.052452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.052412 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.052452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.052437 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.052602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.052580 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.052642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.052580 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.054085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.054064 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.054398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.054378 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.083912 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.083880 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk"] Nov 02 03:13:51.107869 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.107837 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.108176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.108156 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgk68\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cjfrk\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.194710 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.194626 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:51.638738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.638699 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk"] Nov 02 03:13:51.679462 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:13:51.679431 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode524d49f_5df9_48a0_8218_ada7e5466962.slice/crio-228f7ee799465849e3e278b25732874b54a024ddbf5aea2af46bbfc0f60b7e13 WatchSource:0}: Error finding container 228f7ee799465849e3e278b25732874b54a024ddbf5aea2af46bbfc0f60b7e13: Status 404 returned error can't find the container with id 228f7ee799465849e3e278b25732874b54a024ddbf5aea2af46bbfc0f60b7e13 Nov 02 03:13:51.681821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.681784 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:13:51.681931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.681871 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:13:51.681931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:51.681917 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:13:52.276601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:52.276570 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" event={"ID":"e524d49f-5df9-48a0-8218-ada7e5466962","Type":"ContainerStarted","Data":"8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434"} Nov 02 03:13:52.276985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:52.276620 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" event={"ID":"e524d49f-5df9-48a0-8218-ada7e5466962","Type":"ContainerStarted","Data":"228f7ee799465849e3e278b25732874b54a024ddbf5aea2af46bbfc0f60b7e13"} Nov 02 03:13:52.447820 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:52.447763 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podStartSLOduration=2.447748988 podStartE2EDuration="2.447748988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:13:52.445145484 +0000 UTC m=+1574.440846426" watchObservedRunningTime="2026-11-02 03:13:52.447748988 +0000 UTC m=+1574.443449931" Nov 02 03:13:53.195213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:53.195179 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:13:53.197009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:53.196983 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:53.197138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:53.197033 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:13:54.196001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:54.195966 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:54.196403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:54.196023 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:13:55.195078 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:55.195040 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:55.195258 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:55.195102 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:13:56.195266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:56.195229 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:56.195675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:56.195298 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:13:57.195880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:57.195845 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:57.196266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:57.195927 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:13:58.195696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:58.195668 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:58.195861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:58.195715 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:13:59.195586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:59.195553 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 03:13:59.195958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:13:59.195617 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 03:14:00.259501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:00.259464 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:14:00.260000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:00.259789 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:14:00.260631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:00.260614 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:14:03.445131 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:14:03.445094 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:14:05.977927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:05.977891 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h"] Nov 02 03:14:05.981616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:05.981599 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.086192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.086156 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h"] Nov 02 03:14:06.109714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.109689 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.109865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.109721 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.109865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.109789 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.109865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.109857 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.109984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.109939 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.110032 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.110004 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.110072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.110037 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.110072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.110055 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.110205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.110185 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fg8\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211219 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211253 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211271 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211312 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fg8\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211347 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211394 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211424 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211439 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211476 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211733 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211860 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211868 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.211917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.211906 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.212113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.212092 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.213817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.213799 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.214015 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.213992 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.267479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.267412 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.267592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.267578 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fg8\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5552h\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.295345 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.295325 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:06.714255 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.714232 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h"] Nov 02 03:14:06.716438 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:14:06.716408 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175bc536_a8fa_45fa_ad57_6858060cf3d4.slice/crio-a3bed5ff71f12735f5ce3bd979423548dedeb507579d8b735551886638185021 WatchSource:0}: Error finding container a3bed5ff71f12735f5ce3bd979423548dedeb507579d8b735551886638185021: Status 404 returned error can't find the container with id a3bed5ff71f12735f5ce3bd979423548dedeb507579d8b735551886638185021 Nov 02 03:14:06.718593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.718563 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:14:06.718655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.718627 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:14:06.718691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:06.718656 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:14:07.362651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:07.362618 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" event={"ID":"175bc536-a8fa-45fa-ad57-6858060cf3d4","Type":"ContainerStarted","Data":"8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f"} Nov 02 03:14:07.363078 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:07.362669 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" event={"ID":"175bc536-a8fa-45fa-ad57-6858060cf3d4","Type":"ContainerStarted","Data":"a3bed5ff71f12735f5ce3bd979423548dedeb507579d8b735551886638185021"} Nov 02 03:14:07.535381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:07.531437 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podStartSLOduration=2.53141901 podStartE2EDuration="2.53141901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:07.529446412 +0000 UTC m=+1589.525147356" watchObservedRunningTime="2026-11-02 03:14:07.53141901 +0000 UTC m=+1589.527119953" Nov 02 03:14:08.296192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:08.296160 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:08.297754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:08.297721 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 03:14:08.297874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:08.297779 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 03:14:09.295929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:09.295897 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 03:14:09.296365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:09.295951 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 03:14:10.296412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:10.296380 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 03:14:10.296828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:10.296434 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 03:14:11.295943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:11.295909 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 03:14:11.296125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:11.295964 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 03:14:12.295741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:12.295704 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 03:14:12.296133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:12.295775 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 03:14:13.296559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:13.296524 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 03:14:13.296951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:13.296624 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 03:14:15.048944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:15.048911 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:15.049467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:15.049441 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:15.050103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:15.050086 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:14:18.734634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.734600 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5"] Nov 02 03:14:18.738246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.738223 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.757718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.757692 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.757809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.757736 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.757809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.757769 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst8n\" (UniqueName: \"kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.757890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.757823 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.757925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.757897 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.757957 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.757935 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.797295 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.797269 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5"] Nov 02 03:14:18.843642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.843609 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f6e39\"/\"distinct-w0-1f6e39-kserve-self-signed-certs\"" Nov 02 03:14:18.858787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.858758 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.858907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.858837 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.858907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.858858 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.858907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.858878 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst8n\" (UniqueName: \"kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.859051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.858910 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.859051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.858955 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.859273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.859247 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.859378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.859261 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.859440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.859339 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.861117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.861098 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.861436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.861418 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:18.936564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:18.936495 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst8n\" (UniqueName: \"kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n\") pod \"distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:19.050754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:19.050668 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:19.500305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:19.500281 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5"] Nov 02 03:14:19.502343 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:14:19.502316 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58dbe69a_6b18_4d59_8b0a_b0a7c48fb413.slice/crio-14ac34c29599d9a43b6ed09a98d447cad67266c23282ea44613921d95ac119e5 WatchSource:0}: Error finding container 14ac34c29599d9a43b6ed09a98d447cad67266c23282ea44613921d95ac119e5: Status 404 returned error can't find the container with id 14ac34c29599d9a43b6ed09a98d447cad67266c23282ea44613921d95ac119e5 Nov 02 03:14:20.430645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:20.430606 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" event={"ID":"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413","Type":"ContainerStarted","Data":"d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523"} Nov 02 03:14:20.430645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:20.430647 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" event={"ID":"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413","Type":"ContainerStarted","Data":"14ac34c29599d9a43b6ed09a98d447cad67266c23282ea44613921d95ac119e5"} Nov 02 03:14:20.543728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:20.543669 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podStartSLOduration=2.543652937 podStartE2EDuration="2.543652937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:20.543033365 +0000 UTC m=+1602.538734309" watchObservedRunningTime="2026-11-02 03:14:20.543652937 +0000 UTC m=+1602.539353885" Nov 02 03:14:21.050909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:21.050876 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:29.051818 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:29.051780 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:29.063934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:29.063912 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:29.514690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:29.514664 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:14:33.446434 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:14:33.446392 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:14:38.309856 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:38.309799 4868 scope.go:122] "RemoveContainer" containerID="ed3c6ee0e6652623dbe3c4191ae03b24ef84f259bb367683c8bc772d74739815" Nov 02 03:14:53.346906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.346878 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d"] Nov 02 03:14:53.351725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.351707 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.453203 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.453172 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"e2e-conflict-b-5eca459f-kserve-self-signed-certs\"" Nov 02 03:14:53.455259 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.455239 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d"] Nov 02 03:14:53.477204 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.477176 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.477340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.477325 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.477443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.477379 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.477443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.477430 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbr6\" (UniqueName: \"kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.477521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.477494 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.477559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.477541 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578447 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578404 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578552 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578589 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578641 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbr6\" (UniqueName: \"kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578670 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578779 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578720 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.578852 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.578816 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.579126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.579101 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.579229 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.579102 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.580981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.580954 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.581134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.581115 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.635017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.634988 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbr6\" (UniqueName: \"kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6\") pod \"e2e-conflict-b-5eca459f-kserve-885995679-gvj9d\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:53.663145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:53.663115 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:14:54.042503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:54.042472 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d"] Nov 02 03:14:54.669537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:54.669504 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" event={"ID":"c37dff45-c9f4-4f2c-bb2d-b46644bba992","Type":"ContainerStarted","Data":"9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913"} Nov 02 03:14:54.670040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:54.669550 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" event={"ID":"c37dff45-c9f4-4f2c-bb2d-b46644bba992","Type":"ContainerStarted","Data":"df01b41ddf10b038232cc6114bd76248bcd16346163a21c88abed62e4ae30d52"} Nov 02 03:14:54.788431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:54.788374 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" podStartSLOduration=1.788340597 podStartE2EDuration="1.788340597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:14:54.786420043 +0000 UTC m=+1636.782121011" watchObservedRunningTime="2026-11-02 03:14:54.788340597 +0000 UTC m=+1636.784041534" Nov 02 03:14:55.663914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:14:55.663884 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:15:00.167336 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.167300 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr"] Nov 02 03:15:00.171107 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.171086 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl"] Nov 02 03:15:00.171228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.171195 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.174898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.174877 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.236304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.236268 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:15:00.242875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.242831 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bmr\" (UniqueName: \"kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr\") pod \"maas-api-key-cleanup-29893155-c5dnr\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.243163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.243145 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893155-c5dnr\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.298488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.298401 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl"] Nov 02 03:15:00.304060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.304027 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr"] Nov 02 03:15:00.344382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.344328 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.344739 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.344714 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bmr\" (UniqueName: \"kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr\") pod \"maas-api-key-cleanup-29893155-c5dnr\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.344906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.344888 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893155-c5dnr\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.345057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.345038 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmxh\" (UniqueName: \"kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.345853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.345822 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893155-c5dnr\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.401766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.401730 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bmr\" (UniqueName: \"kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr\") pod \"maas-api-key-cleanup-29893155-c5dnr\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.446204 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.446133 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.446204 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.446195 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmxh\" (UniqueName: \"kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.447037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.446993 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.486131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.486103 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:00.587375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.585055 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmxh\" (UniqueName: \"kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.792108 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.792018 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:00.992843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:00.992796 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr"] Nov 02 03:15:01.005642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:01.004562 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:15:01.235378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:01.232294 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl"] Nov 02 03:15:01.717741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:01.717686 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" event={"ID":"0f8040e1-ff27-4500-8ca5-2a832c78c069","Type":"ContainerStarted","Data":"70dcb8e4aa1fe66e0ea12bb53c264d94ad0204c94a065b56092b70e4bf4dc676"} Nov 02 03:15:01.719569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:01.719536 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" event={"ID":"b2a6c0d4-3b69-4109-82d0-9e3851e6af29","Type":"ContainerStarted","Data":"cc9a0d7e7eac34277353ade29392d1bb44387d93a86f091a1959c027e25c791a"} Nov 02 03:15:03.384391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:03.383446 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:15:03.637068 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:03.636946 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:15:03.666184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:03.665411 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:15:03.730677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:03.730649 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:15:03.889418 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:03.878054 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:15:06.811334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:06.811304 4868 generic.go:356] "Generic (PLEG): container finished" podID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerID="3319ac401d0fb33a08dd2d0a8e7d45854bd4097687ceaf21f695f13f930bd0ba" exitCode=60 Nov 02 03:15:06.811763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:06.811390 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" event={"ID":"0f8040e1-ff27-4500-8ca5-2a832c78c069","Type":"ContainerDied","Data":"3319ac401d0fb33a08dd2d0a8e7d45854bd4097687ceaf21f695f13f930bd0ba"} Nov 02 03:15:06.811763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:06.811607 4868 scope.go:122] "RemoveContainer" containerID="3319ac401d0fb33a08dd2d0a8e7d45854bd4097687ceaf21f695f13f930bd0ba" Nov 02 03:15:06.813141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:06.813119 4868 generic.go:356] "Generic (PLEG): container finished" podID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerID="e1b59c2ccee22adba62ad2bd44749eb8d21d6fa70d60da856d28615258e729a8" exitCode=60 Nov 02 03:15:06.813245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:06.813187 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" event={"ID":"b2a6c0d4-3b69-4109-82d0-9e3851e6af29","Type":"ContainerDied","Data":"e1b59c2ccee22adba62ad2bd44749eb8d21d6fa70d60da856d28615258e729a8"} Nov 02 03:15:06.813501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:06.813484 4868 scope.go:122] "RemoveContainer" containerID="e1b59c2ccee22adba62ad2bd44749eb8d21d6fa70d60da856d28615258e729a8" Nov 02 03:15:07.820229 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.820161 4868 generic.go:356] "Generic (PLEG): container finished" podID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerID="d3123776056e1ff279b74d647fa2a1201b038428905d491a7288f742da5e11de" exitCode=60 Nov 02 03:15:07.820659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.820245 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" event={"ID":"0f8040e1-ff27-4500-8ca5-2a832c78c069","Type":"ContainerDied","Data":"d3123776056e1ff279b74d647fa2a1201b038428905d491a7288f742da5e11de"} Nov 02 03:15:07.820659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.820330 4868 scope.go:122] "RemoveContainer" containerID="3319ac401d0fb33a08dd2d0a8e7d45854bd4097687ceaf21f695f13f930bd0ba" Nov 02 03:15:07.820659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.820563 4868 scope.go:122] "RemoveContainer" containerID="d3123776056e1ff279b74d647fa2a1201b038428905d491a7288f742da5e11de" Nov 02 03:15:07.820851 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:07.820830 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893155-c5dnr_odh-ai-gateway-infra(0f8040e1-ff27-4500-8ca5-2a832c78c069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" Nov 02 03:15:07.822246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.822214 4868 generic.go:356] "Generic (PLEG): container finished" podID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerID="7a95756d07a19ccfb48a96b9ebb3200652ede13c0cd781dea843d1037b433b88" exitCode=60 Nov 02 03:15:07.822335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.822259 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" event={"ID":"b2a6c0d4-3b69-4109-82d0-9e3851e6af29","Type":"ContainerDied","Data":"7a95756d07a19ccfb48a96b9ebb3200652ede13c0cd781dea843d1037b433b88"} Nov 02 03:15:07.822485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.822473 4868 scope.go:122] "RemoveContainer" containerID="7a95756d07a19ccfb48a96b9ebb3200652ede13c0cd781dea843d1037b433b88" Nov 02 03:15:07.822686 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:07.822671 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl_odh-ai-gateway-infra(b2a6c0d4-3b69-4109-82d0-9e3851e6af29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" Nov 02 03:15:07.831633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:07.831615 4868 scope.go:122] "RemoveContainer" containerID="e1b59c2ccee22adba62ad2bd44749eb8d21d6fa70d60da856d28615258e729a8" Nov 02 03:15:08.829622 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:08.829594 4868 scope.go:122] "RemoveContainer" containerID="7a95756d07a19ccfb48a96b9ebb3200652ede13c0cd781dea843d1037b433b88" Nov 02 03:15:08.830082 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:08.829858 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl_odh-ai-gateway-infra(b2a6c0d4-3b69-4109-82d0-9e3851e6af29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" Nov 02 03:15:08.831189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:08.831171 4868 scope.go:122] "RemoveContainer" containerID="d3123776056e1ff279b74d647fa2a1201b038428905d491a7288f742da5e11de" Nov 02 03:15:08.831378 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:08.831345 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893155-c5dnr_odh-ai-gateway-infra(0f8040e1-ff27-4500-8ca5-2a832c78c069)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" Nov 02 03:15:18.110206 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:18.110164 4868 scope.go:122] "RemoveContainer" containerID="7a95756d07a19ccfb48a96b9ebb3200652ede13c0cd781dea843d1037b433b88" Nov 02 03:15:18.887380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:18.886955 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" event={"ID":"b2a6c0d4-3b69-4109-82d0-9e3851e6af29","Type":"ContainerStarted","Data":"fa744508cfd87e010e65287d5554c12338f34c63cfe3d5bcb2faa600f026d25c"} Nov 02 03:15:19.000309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.000240 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" podStartSLOduration=14.533867598 podStartE2EDuration="19.000223448s" totalImagesPullingTime="4.46635585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:18.999090335 +0000 UTC m=+1660.994791279" watchObservedRunningTime="2026-11-02 03:15:19.000223448 +0000 UTC m=+1660.995924409" Nov 02 03:15:19.108155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.108123 4868 scope.go:122] "RemoveContainer" containerID="d3123776056e1ff279b74d647fa2a1201b038428905d491a7288f742da5e11de" Nov 02 03:15:19.919376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.919041 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" event={"ID":"0f8040e1-ff27-4500-8ca5-2a832c78c069","Type":"ContainerStarted","Data":"4138fbe7305a9c129e3d396c53295f75b04ae646c2a6f09cc20b7d6c8e1b622a"} Nov 02 03:15:19.926374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.922838 4868 generic.go:356] "Generic (PLEG): container finished" podID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerID="fa744508cfd87e010e65287d5554c12338f34c63cfe3d5bcb2faa600f026d25c" exitCode=60 Nov 02 03:15:19.926374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.922964 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" event={"ID":"b2a6c0d4-3b69-4109-82d0-9e3851e6af29","Type":"ContainerDied","Data":"fa744508cfd87e010e65287d5554c12338f34c63cfe3d5bcb2faa600f026d25c"} Nov 02 03:15:19.926374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.923014 4868 scope.go:122] "RemoveContainer" containerID="7a95756d07a19ccfb48a96b9ebb3200652ede13c0cd781dea843d1037b433b88" Nov 02 03:15:19.936381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:19.931501 4868 scope.go:122] "RemoveContainer" containerID="fa744508cfd87e010e65287d5554c12338f34c63cfe3d5bcb2faa600f026d25c" Nov 02 03:15:19.936381 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:19.931890 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl_odh-ai-gateway-infra(b2a6c0d4-3b69-4109-82d0-9e3851e6af29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" Nov 02 03:15:20.026375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:20.026256 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl"] Nov 02 03:15:20.206758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:20.206620 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" podStartSLOduration=15.319848462 podStartE2EDuration="20.206601738s" totalImagesPullingTime="4.886753276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:20.079142637 +0000 UTC m=+1662.074843581" watchObservedRunningTime="2026-11-02 03:15:20.206601738 +0000 UTC m=+1662.202302677" Nov 02 03:15:21.082376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.062327 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr"] Nov 02 03:15:21.188500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.188146 4868 generic.go:356] "Generic (PLEG): container finished" podID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerID="4138fbe7305a9c129e3d396c53295f75b04ae646c2a6f09cc20b7d6c8e1b622a" exitCode=60 Nov 02 03:15:21.188500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.188214 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" event={"ID":"0f8040e1-ff27-4500-8ca5-2a832c78c069","Type":"ContainerDied","Data":"4138fbe7305a9c129e3d396c53295f75b04ae646c2a6f09cc20b7d6c8e1b622a"} Nov 02 03:15:21.188500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.188269 4868 scope.go:122] "RemoveContainer" containerID="d3123776056e1ff279b74d647fa2a1201b038428905d491a7288f742da5e11de" Nov 02 03:15:21.810133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.809815 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:21.870408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.870384 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:21.879201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.879177 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca\") pod \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " Nov 02 03:15:21.879364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.879234 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh\" (UniqueName: \"kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh\") pod \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\" (UID: \"b2a6c0d4-3b69-4109-82d0-9e3851e6af29\") " Nov 02 03:15:21.879620 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.879591 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca" pod "b2a6c0d4-3b69-4109-82d0-9e3851e6af29" (UID: "b2a6c0d4-3b69-4109-82d0-9e3851e6af29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:15:21.881202 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.881177 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh" pod "b2a6c0d4-3b69-4109-82d0-9e3851e6af29" (UID: "b2a6c0d4-3b69-4109-82d0-9e3851e6af29"). InnerVolumeSpecName "kube-api-access-djmxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:15:21.980612 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.980582 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr\" (UniqueName: \"kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr\") pod \"0f8040e1-ff27-4500-8ca5-2a832c78c069\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " Nov 02 03:15:21.980799 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.980694 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca\") pod \"0f8040e1-ff27-4500-8ca5-2a832c78c069\" (UID: \"0f8040e1-ff27-4500-8ca5-2a832c78c069\") " Nov 02 03:15:21.980963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.980941 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:15:21.980963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.980959 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmxh\" (UniqueName: \"kubernetes.io/projected/b2a6c0d4-3b69-4109-82d0-9e3851e6af29-kube-api-access-djmxh\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:15:21.981134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.981005 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca" pod "0f8040e1-ff27-4500-8ca5-2a832c78c069" (UID: "0f8040e1-ff27-4500-8ca5-2a832c78c069"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:15:21.982733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:21.982712 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr" pod "0f8040e1-ff27-4500-8ca5-2a832c78c069" (UID: "0f8040e1-ff27-4500-8ca5-2a832c78c069"). InnerVolumeSpecName "kube-api-access-z8bmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:15:22.082217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.082139 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8bmr\" (UniqueName: \"kubernetes.io/projected/0f8040e1-ff27-4500-8ca5-2a832c78c069-kube-api-access-z8bmr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:15:22.082217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.082171 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8040e1-ff27-4500-8ca5-2a832c78c069-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:15:22.194299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.194276 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" Nov 02 03:15:22.194716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.194329 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr" event={"ID":"0f8040e1-ff27-4500-8ca5-2a832c78c069","Type":"ContainerDied","Data":"70dcb8e4aa1fe66e0ea12bb53c264d94ad0204c94a065b56092b70e4bf4dc676"} Nov 02 03:15:22.194716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.194402 4868 scope.go:122] "RemoveContainer" containerID="4138fbe7305a9c129e3d396c53295f75b04ae646c2a6f09cc20b7d6c8e1b622a" Nov 02 03:15:22.195719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.195691 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" event={"ID":"b2a6c0d4-3b69-4109-82d0-9e3851e6af29","Type":"ContainerDied","Data":"cc9a0d7e7eac34277353ade29392d1bb44387d93a86f091a1959c027e25c791a"} Nov 02 03:15:22.195799 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.195694 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl" Nov 02 03:15:22.202975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.202949 4868 scope.go:122] "RemoveContainer" containerID="fa744508cfd87e010e65287d5554c12338f34c63cfe3d5bcb2faa600f026d25c" Nov 02 03:15:22.358761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.357787 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr"] Nov 02 03:15:22.374155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.361344 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893155-c5dnr"] Nov 02 03:15:22.468206 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.468180 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl"] Nov 02 03:15:22.475803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:22.475776 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893155-kplvl"] Nov 02 03:15:24.113765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:24.113728 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" path="/var/lib/kubelet/pods/0f8040e1-ff27-4500-8ca5-2a832c78c069/volumes" Nov 02 03:15:24.114145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:24.114104 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" path="/var/lib/kubelet/pods/b2a6c0d4-3b69-4109-82d0-9e3851e6af29/volumes" Nov 02 03:15:30.972440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.972406 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t"] Nov 02 03:15:30.972998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.972978 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.973076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973001 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.973076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973020 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:30.973076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973030 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:30.973076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973057 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:30.973076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973065 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973099 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973107 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973218 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973232 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973242 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973253 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:30.973321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.973264 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:30.976044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:30.976024 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001519 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001663 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001578 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001679 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc78t\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001706 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001738 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001785 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001886 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.001956 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.001945 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.002029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.002017 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.102995 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103194 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103049 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103194 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103075 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103194 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103168 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103230 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103270 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc78t\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103289 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103386 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103417 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103449 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103479 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103676 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.103865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.103844 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.104165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.104144 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.105536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.105516 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.105638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.105591 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.161603 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.161581 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.162183 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.162160 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc78t\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.173042 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.173020 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t"] Nov 02 03:15:31.289261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.289184 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:31.715745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.715711 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t"] Nov 02 03:15:31.725312 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:15:31.725274 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a974a34_3184_4677_9893_d32e4193d600.slice/crio-c9d52039ff3b8156e2fb0ac985a4cbd1f6d4ee13108072997778a5912b2354b9 WatchSource:0}: Error finding container c9d52039ff3b8156e2fb0ac985a4cbd1f6d4ee13108072997778a5912b2354b9: Status 404 returned error can't find the container with id c9d52039ff3b8156e2fb0ac985a4cbd1f6d4ee13108072997778a5912b2354b9 Nov 02 03:15:31.743450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.742056 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:15:31.743450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.742170 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:15:31.743450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:31.742218 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:15:32.268077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:32.268044 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" event={"ID":"2a974a34-3184-4677-9893-d32e4193d600","Type":"ContainerStarted","Data":"1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42"} Nov 02 03:15:32.268571 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:32.268096 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" event={"ID":"2a974a34-3184-4677-9893-d32e4193d600","Type":"ContainerStarted","Data":"c9d52039ff3b8156e2fb0ac985a4cbd1f6d4ee13108072997778a5912b2354b9"} Nov 02 03:15:32.289476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:32.289442 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:32.291280 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:32.291253 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:32.291417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:32.291301 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:32.482375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:32.477504 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podStartSLOduration=2.477481492 podStartE2EDuration="2.477481492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:32.43307667 +0000 UTC m=+1674.428777614" watchObservedRunningTime="2026-11-02 03:15:32.477481492 +0000 UTC m=+1674.473182436" Nov 02 03:15:33.290184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:33.290152 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:33.290595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:33.290206 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:33.471131 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:15:33.471095 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:15:34.289886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:34.289852 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:34.290054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:34.289912 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:35.290027 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:35.289991 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:35.290547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:35.290061 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:36.289819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:36.289782 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:36.290001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:36.289843 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:37.290103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.290067 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:37.290531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.290130 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:37.657259 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.657218 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h"] Nov 02 03:15:37.657916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.657894 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:37.657967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.657918 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:37.658013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.657973 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:37.658013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.657982 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8040e1-ff27-4500-8ca5-2a832c78c069" containerName="cleanup" Nov 02 03:15:37.658105 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.658092 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a6c0d4-3b69-4109-82d0-9e3851e6af29" containerName="cleanup" Nov 02 03:15:37.666542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.666522 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.694708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.694671 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.694882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.694858 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.694941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.694912 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.695032 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.695016 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.695077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.695048 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.695077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.695066 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dr9\" (UniqueName: \"kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.710879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.710854 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h"] Nov 02 03:15:37.757835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.757805 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"openshift-service-ca.crt\"" Nov 02 03:15:37.758047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.757901 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"kube-root-ca.crt\"" Nov 02 03:15:37.768598 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.768575 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"default-dockercfg-n7lmt\"" Nov 02 03:15:37.768812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.768794 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-398828\"/\"distinct-w4-398828-kserve-self-signed-certs\"" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.798040 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.798124 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.798205 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.798240 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.798270 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dr9\" (UniqueName: \"kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.798320 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.799223 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.799949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.799514 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.806445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.805673 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.806445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.806221 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.821373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.815996 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.878517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.872098 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dr9\" (UniqueName: \"kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9\") pod \"distinct-w4-398828-kserve-6574c7d75-zhx8h\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:37.979854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:37.979762 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:38.289969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:38.289885 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 02 03:15:38.289969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:38.289937 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 02 03:15:38.485841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:38.485810 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h"] Nov 02 03:15:38.486211 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:15:38.486190 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71d6712e_e0c6_4897_9c03_65546a869c0a.slice/crio-51dd5d32872a45b5f56f5bed6c7f1ee3795189f49ff6eb8bc9c88ec6cb61737a WatchSource:0}: Error finding container 51dd5d32872a45b5f56f5bed6c7f1ee3795189f49ff6eb8bc9c88ec6cb61737a: Status 404 returned error can't find the container with id 51dd5d32872a45b5f56f5bed6c7f1ee3795189f49ff6eb8bc9c88ec6cb61737a Nov 02 03:15:39.317824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.317784 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" event={"ID":"71d6712e-e0c6-4897-9c03-65546a869c0a","Type":"ContainerStarted","Data":"13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497"} Nov 02 03:15:39.317824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.317827 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" event={"ID":"71d6712e-e0c6-4897-9c03-65546a869c0a","Type":"ContainerStarted","Data":"51dd5d32872a45b5f56f5bed6c7f1ee3795189f49ff6eb8bc9c88ec6cb61737a"} Nov 02 03:15:39.343118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.343023 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:39.343627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.343590 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:39.349071 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.346186 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:15:39.430888 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.430835 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podStartSLOduration=2.430818108 podStartE2EDuration="2.430818108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:39.429639863 +0000 UTC m=+1681.425340806" watchObservedRunningTime="2026-11-02 03:15:39.430818108 +0000 UTC m=+1681.426519051" Nov 02 03:15:39.980457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:39.980421 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:47.548024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.547991 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r"] Nov 02 03:15:47.551830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.551814 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.604930 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.604895 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.605058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.604960 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.605119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.605084 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.605247 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.605225 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.605404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.605379 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.605805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.605780 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wczw\" (UniqueName: \"kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.654461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.654435 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"default-dockercfg-4jt6g\"" Nov 02 03:15:47.654590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.654444 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"openshift-service-ca.crt\"" Nov 02 03:15:47.654590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.654476 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"distinct-2-w1-5ac93f-kserve-self-signed-certs\"" Nov 02 03:15:47.654681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.654624 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5ac93f\"/\"kube-root-ca.crt\"" Nov 02 03:15:47.656486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.656467 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r"] Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709009 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709163 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709208 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709276 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wczw\" (UniqueName: \"kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709315 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709414 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.710374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.709909 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.734385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.713625 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.734385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.727634 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.737994 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.737253 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.740064 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.739984 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.769492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.769453 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wczw\" (UniqueName: \"kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw\") pod \"distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.863660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.863629 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:47.986627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:47.985748 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:48.064777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:48.064496 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:48.357926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:48.357903 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r"] Nov 02 03:15:48.358771 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:15:48.358748 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a4763e_0808_463d_9a43_0b7e20ff7aa7.slice/crio-512617c59f70e5f08038b7d3ed894e4b2732e1e4b27259dffec1b756c28b3428 WatchSource:0}: Error finding container 512617c59f70e5f08038b7d3ed894e4b2732e1e4b27259dffec1b756c28b3428: Status 404 returned error can't find the container with id 512617c59f70e5f08038b7d3ed894e4b2732e1e4b27259dffec1b756c28b3428 Nov 02 03:15:48.378109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:48.378086 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:15:49.371156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:49.371121 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" event={"ID":"62a4763e-0808-463d-9a43-0b7e20ff7aa7","Type":"ContainerStarted","Data":"41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd"} Nov 02 03:15:49.371625 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:49.371164 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" event={"ID":"62a4763e-0808-463d-9a43-0b7e20ff7aa7","Type":"ContainerStarted","Data":"512617c59f70e5f08038b7d3ed894e4b2732e1e4b27259dffec1b756c28b3428"} Nov 02 03:15:49.484880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:49.484831 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podStartSLOduration=2.484817548 podStartE2EDuration="2.484817548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:15:49.483180158 +0000 UTC m=+1691.478881111" watchObservedRunningTime="2026-11-02 03:15:49.484817548 +0000 UTC m=+1691.480518540" Nov 02 03:15:49.864204 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:49.864175 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:57.864156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:57.864123 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:57.876268 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:57.876244 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:15:58.438169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:15:58.438136 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:16:03.447966 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:16:03.447928 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:16:03.905472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:03.905440 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d"] Nov 02 03:16:15.372414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:15.372131 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 03:16:15.512386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:15.504267 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 02 03:16:15.572556 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:15.572507 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:15.814432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:15.813461 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:16.113839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.113466 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:16:16.113839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.113545 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:16.235382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.234474 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:16.325528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.325479 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:16:16.325870 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.325834 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:16.333638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.333598 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 02 03:16:16.635377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.634486 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:16:16.635377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.634567 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:16.840075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.837588 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:16.840075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:16.838384 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:16:17.359706 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:17.359672 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:16:18.933553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:18.933514 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerName="main" containerID="cri-o://9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913" gracePeriod=45 Nov 02 03:16:19.199693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.199637 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:16:19.244237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244204 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs\" (UniqueName: \"kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs\") pod \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " Nov 02 03:16:19.244408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244243 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6\" (UniqueName: \"kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6\") pod \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " Nov 02 03:16:19.244408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244294 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache\") pod \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " Nov 02 03:16:19.244540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244411 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home\") pod \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " Nov 02 03:16:19.244540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244481 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir\") pod \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " Nov 02 03:16:19.244638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244541 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm\") pod \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\" (UID: \"c37dff45-c9f4-4f2c-bb2d-b46644bba992\") " Nov 02 03:16:19.244638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244569 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache" pod "c37dff45-c9f4-4f2c-bb2d-b46644bba992" (UID: "c37dff45-c9f4-4f2c-bb2d-b46644bba992"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:16:19.244740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244668 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home" pod "c37dff45-c9f4-4f2c-bb2d-b46644bba992" (UID: "c37dff45-c9f4-4f2c-bb2d-b46644bba992"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:16:19.244740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244696 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir" pod "c37dff45-c9f4-4f2c-bb2d-b46644bba992" (UID: "c37dff45-c9f4-4f2c-bb2d-b46644bba992"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:16:19.244990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244966 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:16:19.244990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.244989 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:16:19.245167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.245000 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:16:19.246511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.246489 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm" pod "c37dff45-c9f4-4f2c-bb2d-b46644bba992" (UID: "c37dff45-c9f4-4f2c-bb2d-b46644bba992"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:16:19.246593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.246530 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs" pod "c37dff45-c9f4-4f2c-bb2d-b46644bba992" (UID: "c37dff45-c9f4-4f2c-bb2d-b46644bba992"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:16:19.246593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.246567 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6" pod "c37dff45-c9f4-4f2c-bb2d-b46644bba992" (UID: "c37dff45-c9f4-4f2c-bb2d-b46644bba992"). InnerVolumeSpecName "kube-api-access-jjbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:16:19.345653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.345622 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c37dff45-c9f4-4f2c-bb2d-b46644bba992-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:16:19.345653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.345650 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c37dff45-c9f4-4f2c-bb2d-b46644bba992-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:16:19.345840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.345663 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjbr6\" (UniqueName: \"kubernetes.io/projected/c37dff45-c9f4-4f2c-bb2d-b46644bba992-kube-api-access-jjbr6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:16:19.887597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.887567 4868 generic.go:356] "Generic (PLEG): container finished" podID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerID="9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913" exitCode=0 Nov 02 03:16:19.887765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.887611 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" Nov 02 03:16:19.887765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.887655 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" event={"ID":"c37dff45-c9f4-4f2c-bb2d-b46644bba992","Type":"ContainerDied","Data":"9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913"} Nov 02 03:16:19.887765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.887700 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d" event={"ID":"c37dff45-c9f4-4f2c-bb2d-b46644bba992","Type":"ContainerDied","Data":"df01b41ddf10b038232cc6114bd76248bcd16346163a21c88abed62e4ae30d52"} Nov 02 03:16:19.887765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.887723 4868 scope.go:122] "RemoveContainer" containerID="9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913" Nov 02 03:16:19.897272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.897253 4868 scope.go:122] "RemoveContainer" containerID="9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913" Nov 02 03:16:19.897538 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:16:19.897521 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913\": container with ID starting with 9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913 not found: ID does not exist" containerID="9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913" Nov 02 03:16:19.897600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:19.897561 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913"} err="failed to get container status \"9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913\": rpc error: code = NotFound desc = could not find container \"9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913\": container with ID starting with 9d50ab2ae902e11256fc0e28aca80b21d89a22f8d150babb64bf19e8102ac913 not found: ID does not exist" Nov 02 03:16:20.048843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:20.048818 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d"] Nov 02 03:16:20.052192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:20.052166 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/e2e-conflict-b-5eca459f-kserve-885995679-gvj9d"] Nov 02 03:16:20.118540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:20.118505 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" path="/var/lib/kubelet/pods/c37dff45-c9f4-4f2c-bb2d-b46644bba992/volumes" Nov 02 03:16:27.367515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.367431 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:27.429475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.429426 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:27.493497 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.493437 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:27.504482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.504428 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:27.544497 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.544450 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:27.880377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.876464 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:16:27.880377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.876542 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:27.921537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:27.921434 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:28.327559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:28.327438 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:16:33.446284 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:16:33.446245 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:16:37.052491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.052460 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98"] Nov 02 03:16:37.052903 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.052888 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerName="main" Nov 02 03:16:37.052903 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.052899 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerName="main" Nov 02 03:16:37.052986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.052967 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37dff45-c9f4-4f2c-bb2d-b46644bba992" containerName="main" Nov 02 03:16:37.056150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.056132 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072502 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072641 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072533 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072671 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072717 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072753 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072820 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072878 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072867 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072894 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.072919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.072912 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb8l\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.085742 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.085713 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w"] Nov 02 03:16:37.089166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.089151 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.123389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.114861 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98"] Nov 02 03:16:37.174347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.174303 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w"] Nov 02 03:16:37.175639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175607 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.175763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175656 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.175763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175688 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.175763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175717 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.175763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175742 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.175990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175810 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlhm\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.175990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175840 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.175990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175881 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.175990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175951 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.176195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.175996 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.176195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176031 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.176195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176066 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.176195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176094 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.176195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176129 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.176195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176159 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.176639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176221 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.176639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176281 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.176639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.176313 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb8l\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.178686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.177856 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.180975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.179000 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.180975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.179613 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.181137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.180504 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.181137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.180554 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.183647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.183620 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.183733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.183688 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.271377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.270744 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277056 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb8l\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277683 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277739 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277838 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277870 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277895 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277959 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlhm\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.277990 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.278028 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.278078 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.278475 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.278707 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.279367 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.279603 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.291460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.279865 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.301379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.295803 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.311883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.311773 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.362166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.362124 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.362390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.362222 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlhm\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.369564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.369337 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:37.405453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.405419 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:37.798347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.798307 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98"] Nov 02 03:16:37.835024 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:16:37.834997 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63323a25_54f2_4c30_8154_662079ca62e4.slice/crio-7535232a665e0f43a66362dacb9878119ed2733fee809874a583a28d408c0c86 WatchSource:0}: Error finding container 7535232a665e0f43a66362dacb9878119ed2733fee809874a583a28d408c0c86: Status 404 returned error can't find the container with id 7535232a665e0f43a66362dacb9878119ed2733fee809874a583a28d408c0c86 Nov 02 03:16:37.837209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.837175 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:37.837316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.837279 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:37.837604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.837326 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:37.847314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.847291 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w"] Nov 02 03:16:37.849627 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:16:37.849604 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01832d40_9375_4879_97b0_9e5723711232.slice/crio-9f368a1a77a4d906453332779ba269e40def961dfc1c12c4455be0a43eb7b8df WatchSource:0}: Error finding container 9f368a1a77a4d906453332779ba269e40def961dfc1c12c4455be0a43eb7b8df: Status 404 returned error can't find the container with id 9f368a1a77a4d906453332779ba269e40def961dfc1c12c4455be0a43eb7b8df Nov 02 03:16:37.851952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.851922 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:37.852026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.851996 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:37.852065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:37.852044 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:38.021937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.021902 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" event={"ID":"63323a25-54f2-4c30-8154-662079ca62e4","Type":"ContainerStarted","Data":"2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a"} Nov 02 03:16:38.022121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.021962 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" event={"ID":"63323a25-54f2-4c30-8154-662079ca62e4","Type":"ContainerStarted","Data":"7535232a665e0f43a66362dacb9878119ed2733fee809874a583a28d408c0c86"} Nov 02 03:16:38.024104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.024071 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" event={"ID":"01832d40-9375-4879-97b0-9e5723711232","Type":"ContainerStarted","Data":"35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f"} Nov 02 03:16:38.024273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.024122 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" event={"ID":"01832d40-9375-4879-97b0-9e5723711232","Type":"ContainerStarted","Data":"9f368a1a77a4d906453332779ba269e40def961dfc1c12c4455be0a43eb7b8df"} Nov 02 03:16:38.137665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.137238 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podStartSLOduration=1.1372249110000001 podStartE2EDuration="1.137224911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:38.13652805 +0000 UTC m=+1740.132228994" watchObservedRunningTime="2026-11-02 03:16:38.137224911 +0000 UTC m=+1740.132925930" Nov 02 03:16:38.259641 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.259576 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podStartSLOduration=1.259557388 podStartE2EDuration="1.259557388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:38.25683405 +0000 UTC m=+1740.252535022" watchObservedRunningTime="2026-11-02 03:16:38.259557388 +0000 UTC m=+1740.255258332" Nov 02 03:16:38.370644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.370614 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:38.376734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.376709 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:38.376840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.376759 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:38.409461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.405977 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:38.410514 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.410477 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:38.410655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:38.410547 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:39.381692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:39.375712 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:39.381692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:39.375776 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:39.406225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:39.406184 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:39.406370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:39.406247 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:40.377860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:40.376736 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:40.377860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:40.376797 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:40.405727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:40.405689 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:40.406122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:40.405766 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:41.382755 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:41.382701 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:41.382954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:41.382768 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:41.406618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:41.406584 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:41.406985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:41.406636 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:42.377376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:42.376667 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:42.377376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:42.376730 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:42.406338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:42.406295 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:42.406519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:42.406347 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:43.375400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:43.374643 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:43.375400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:43.374705 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:43.405837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:43.405801 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:43.406015 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:43.405863 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:44.370401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:44.370344 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:16:44.370600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:44.370430 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:16:44.405884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:44.405849 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:16:44.406274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:44.405913 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:16:45.374484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:45.374453 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:45.374779 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:45.374761 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:45.375487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:45.375467 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:16:45.409432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:45.409413 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:45.409827 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:45.409668 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:45.410339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:45.410322 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:16:50.476983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.476908 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9"] Nov 02 03:16:50.481781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.481761 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.526432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.526406 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9"] Nov 02 03:16:50.529963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.529946 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.580992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.580969 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9"] Nov 02 03:16:50.633886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.633860 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.634197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634120 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634226 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634264 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634286 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634325 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634363 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634483 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634547 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634573 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634596 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634622 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stwwt\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634682 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634706 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634741 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634770 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5kb\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634829 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.635729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.634881 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.637209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.636157 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9"] Nov 02 03:16:50.735700 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735619 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.735700 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735679 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.735905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735704 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.735905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735747 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.735905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735772 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.735905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735804 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.735905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735863 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.735905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735892 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735921 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.735948 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stwwt\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736013 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736036 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736072 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736100 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5kb\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736162 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736214 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736259 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.736610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736297 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.736610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736413 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.736830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736652 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.736889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736872 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.737155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.737124 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.737257 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.737160 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.737397 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.737374 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.737477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.736213 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.737677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.737654 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.737945 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.737923 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.738378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.738320 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.738595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.738570 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.739062 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.739042 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.739757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.739733 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.739890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.739875 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.792821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.792794 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.792943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.792896 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.792997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.792971 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stwwt\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:50.793221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.793202 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5kb\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.794952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.794931 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:50.843247 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:50.843216 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:51.220783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.220745 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9"] Nov 02 03:16:51.260987 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:16:51.260957 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f53e4e_843e_4930_81f9_329875f19c7f.slice/crio-3e65b865c086d011a30b55ffd316bb9076fe2fa13120251939137b0d235a16d3 WatchSource:0}: Error finding container 3e65b865c086d011a30b55ffd316bb9076fe2fa13120251939137b0d235a16d3: Status 404 returned error can't find the container with id 3e65b865c086d011a30b55ffd316bb9076fe2fa13120251939137b0d235a16d3 Nov 02 03:16:51.263228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.263195 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:51.263321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.263263 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:51.263321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.263290 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:51.275748 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:16:51.275707 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937c3c81_800d_4c11_9903_a8881d96162d.slice/crio-0df595036458c56888ffb1849fcfd20b6079b32ad5ba027aab1638f93dc9497e WatchSource:0}: Error finding container 0df595036458c56888ffb1849fcfd20b6079b32ad5ba027aab1638f93dc9497e: Status 404 returned error can't find the container with id 0df595036458c56888ffb1849fcfd20b6079b32ad5ba027aab1638f93dc9497e Nov 02 03:16:51.275849 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.275742 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9"] Nov 02 03:16:51.278033 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.278007 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:51.278126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.278074 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:51.278192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:51.278141 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:16:52.216441 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.216409 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" event={"ID":"937c3c81-800d-4c11-9903-a8881d96162d","Type":"ContainerStarted","Data":"7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002"} Nov 02 03:16:52.216919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.216468 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" event={"ID":"937c3c81-800d-4c11-9903-a8881d96162d","Type":"ContainerStarted","Data":"0df595036458c56888ffb1849fcfd20b6079b32ad5ba027aab1638f93dc9497e"} Nov 02 03:16:52.218414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.218390 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" event={"ID":"12f53e4e-843e-4930-81f9-329875f19c7f","Type":"ContainerStarted","Data":"841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6"} Nov 02 03:16:52.218570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.218437 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" event={"ID":"12f53e4e-843e-4930-81f9-329875f19c7f","Type":"ContainerStarted","Data":"3e65b865c086d011a30b55ffd316bb9076fe2fa13120251939137b0d235a16d3"} Nov 02 03:16:52.408893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.408839 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podStartSLOduration=2.408824591 podStartE2EDuration="2.408824591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:52.406739287 +0000 UTC m=+1754.402440230" watchObservedRunningTime="2026-11-02 03:16:52.408824591 +0000 UTC m=+1754.404525534" Nov 02 03:16:52.525046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.524943 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podStartSLOduration=2.5249297200000003 podStartE2EDuration="2.52492972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:16:52.52325889 +0000 UTC m=+1754.518959834" watchObservedRunningTime="2026-11-02 03:16:52.52492972 +0000 UTC m=+1754.520630759" Nov 02 03:16:52.795605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.795515 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:52.797502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.797475 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 03:16:52.797635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.797519 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 03:16:52.862377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.856904 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:52.862377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.858890 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 03:16:52.862377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:52.858939 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 03:16:53.795813 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:53.795770 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 03:16:53.796319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:53.795867 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 03:16:53.848760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:53.848720 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 03:16:53.849015 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:53.848989 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 03:16:54.796901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:54.796867 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 03:16:54.797406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:54.796939 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 03:16:54.882432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:54.880976 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 03:16:54.882432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:54.881053 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 03:16:55.795852 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:55.795818 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 03:16:55.796015 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:55.795872 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 03:16:55.844702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:55.844667 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 03:16:55.845241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:55.845214 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 03:16:56.795797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:56.795766 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 03:16:56.796010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:56.795987 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 03:16:56.843733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:56.843699 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 03:16:56.843983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:56.843946 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 03:16:57.796069 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:57.796030 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 02 03:16:57.796600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:57.796095 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 02 03:16:57.843923 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:57.843872 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 02 03:16:57.844123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:57.843951 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 02 03:16:58.872860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:58.872813 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:58.875229 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:58.875207 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:58.884091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:58.884062 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:16:58.980544 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:58.980516 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:58.980941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:58.980917 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:16:58.982193 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:16:58.982172 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:17:03.447872 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:17:03.447833 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:17:08.984488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:08.984437 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:09.302533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.302430 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:09.384376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.375432 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:09.425487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.425443 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:09.441974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.441934 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:09.496241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.495450 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 02 03:17:09.510471 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.510426 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:09.540478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:09.540432 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:10.068586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:10.067465 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:15.426325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.426295 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp"] Nov 02 03:17:15.431312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.431290 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.483800 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.483769 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.483983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.483821 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.483983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.483898 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.483983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.483966 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrhc\" (UniqueName: \"kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.484155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.484020 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.484155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.484143 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.487521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.487492 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp"] Nov 02 03:17:15.532953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.532919 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ca6546\"/\"distinct-w2-ca6546-kserve-self-signed-certs\"" Nov 02 03:17:15.591407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.586651 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrhc\" (UniqueName: \"kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.591407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.588588 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.591407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.588938 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.591407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.589000 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.591407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.589031 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.591407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.589128 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.638012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.605246 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.638587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.625830 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.638587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.627208 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.640614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.639132 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.645081 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.645051 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.646683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.646621 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrhc\" (UniqueName: \"kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc\") pod \"distinct-w2-ca6546-kserve-898b79df6-hkfsp\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:15.743602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:15.743527 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:16.120790 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:16.120761 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp"] Nov 02 03:17:16.122154 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:17:16.122131 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6abbaca8_8870_4569_ae4d_4e188d620122.slice/crio-11b74989f33267b36bf870f4e785b140e46fd55d80e8f5ef52ef49ba77a411ce WatchSource:0}: Error finding container 11b74989f33267b36bf870f4e785b140e46fd55d80e8f5ef52ef49ba77a411ce: Status 404 returned error can't find the container with id 11b74989f33267b36bf870f4e785b140e46fd55d80e8f5ef52ef49ba77a411ce Nov 02 03:17:16.484859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:16.484751 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" event={"ID":"6abbaca8-8870-4569-ae4d-4e188d620122","Type":"ContainerStarted","Data":"ad2114f9b898c38439048478db51110f80bbd2316699daaacb7d3e91fef505d0"} Nov 02 03:17:16.484859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:16.484816 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" event={"ID":"6abbaca8-8870-4569-ae4d-4e188d620122","Type":"ContainerStarted","Data":"11b74989f33267b36bf870f4e785b140e46fd55d80e8f5ef52ef49ba77a411ce"} Nov 02 03:17:16.599744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:16.599695 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podStartSLOduration=1.5996811960000001 podStartE2EDuration="1.599681196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:16.597424691 +0000 UTC m=+1778.593125653" watchObservedRunningTime="2026-11-02 03:17:16.599681196 +0000 UTC m=+1778.595382140" Nov 02 03:17:16.753432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:16.750439 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:25.683693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.683656 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf"] Nov 02 03:17:25.687505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.687485 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.739468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.739446 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-398828-openshift-default-dockercfg-dqq8t\"" Nov 02 03:17:25.744114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.744092 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:25.758258 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.758240 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:25.792044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792015 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf"] Nov 02 03:17:25.792457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792435 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.792545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792521 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsm4r\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.792602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792587 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.792645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792630 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.792692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792675 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.792747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792728 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.792939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.792916 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.793041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.793019 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.793103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.793067 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.894677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894648 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.894835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894743 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.894835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894793 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.894835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894825 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.894951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894879 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.894951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894915 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsm4r\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.894964 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895011 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895042 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895251 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895216 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895465 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895500 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895551 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.895700 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.895632 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.897606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.897588 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.897934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.897914 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.950806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.950753 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:25.951137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:25.951112 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsm4r\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-6gsrf\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:26.001051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.001026 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:26.420597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.420571 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf"] Nov 02 03:17:26.422531 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:17:26.422504 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc151c523_e65a_4a2e_b75e_cfee2307c48a.slice/crio-e8ab776d4055adf5998e9e0c1e502bf194ad3f002e80216fa63aafeea4d47978 WatchSource:0}: Error finding container e8ab776d4055adf5998e9e0c1e502bf194ad3f002e80216fa63aafeea4d47978: Status 404 returned error can't find the container with id e8ab776d4055adf5998e9e0c1e502bf194ad3f002e80216fa63aafeea4d47978 Nov 02 03:17:26.424869 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.424840 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:17:26.424941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.424907 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:17:26.424941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.424938 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:17:26.546145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.544865 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" event={"ID":"c151c523-e65a-4a2e-b75e-cfee2307c48a","Type":"ContainerStarted","Data":"35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799"} Nov 02 03:17:26.546145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.544930 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" event={"ID":"c151c523-e65a-4a2e-b75e-cfee2307c48a","Type":"ContainerStarted","Data":"e8ab776d4055adf5998e9e0c1e502bf194ad3f002e80216fa63aafeea4d47978"} Nov 02 03:17:26.563435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.563405 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:26.662959 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:26.662909 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podStartSLOduration=1.662895792 podStartE2EDuration="1.662895792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:26.660256019 +0000 UTC m=+1788.655956960" watchObservedRunningTime="2026-11-02 03:17:26.662895792 +0000 UTC m=+1788.658596729" Nov 02 03:17:27.001988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:27.001962 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:27.011831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:27.011799 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:27.012081 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:27.012052 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:28.003877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.003841 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:28.004535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.004486 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:28.486376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.485515 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:28.537392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.537304 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:28.548488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.548450 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:28.556164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.555424 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:28.572799 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.572467 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:28.875377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:28.868834 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:29.008376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.005334 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:29.008376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.005415 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:29.008376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.005819 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:29.229632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.229533 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:17:29.229910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.229876 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:29.247487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.247437 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Nov 02 03:17:29.326496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.326441 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:29.524437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.485696 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 03:17:29.546214 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.546177 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:17:29.587383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.564552 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:29.587383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.564939 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:17:29.587383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.569425 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:29.587383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.569783 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:29.635068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:29.616877 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:17:30.013521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.013483 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:17:30.254394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.252084 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:30.254394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.071859 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:30.254394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.053855 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:30.254394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.252515 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:30.254394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.094583 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:17:30.254394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.252586 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:17:30.287699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.275719 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Nov 02 03:17:30.348376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.347514 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:17:30.348376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.347578 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:17:30.479292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.479258 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:17:30.575476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.563676 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 03:17:30.696808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:30.696782 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:17:31.001887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:31.001850 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:31.002164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:31.002133 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:32.002406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:32.002343 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:32.002793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:32.002427 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:33.001845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:33.001810 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:33.002016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:33.001868 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:33.462980 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:17:33.462940 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:17:34.001934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:34.001893 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:34.002138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:34.001953 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:35.001691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:35.001655 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 03:17:35.002162 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:35.001732 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 03:17:36.005940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:36.005909 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:36.006418 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:36.006212 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:36.006940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:36.006915 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:17:38.287407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:38.287377 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:17:38.292033 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:38.292010 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:17:46.994237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:46.994203 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw"] Nov 02 03:17:46.999517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:46.999495 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.107344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.107312 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw"] Nov 02 03:17:47.142704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142681 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.142857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142709 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.142857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142728 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrb97\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.142857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142747 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.142978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142852 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.142978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142958 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.143058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.142997 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.143058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.143023 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.143199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.143177 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244590 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244623 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244648 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrb97\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244677 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244708 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244772 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244818 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.244906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244853 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.245157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.244934 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.245157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.245138 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.245344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.245316 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.245466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.245428 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.245628 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.245607 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.245793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.245771 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.247651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.247632 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.247785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.247771 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.300639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.300616 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.300847 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.300827 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrb97\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.312660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.312638 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:47.732946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.732918 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw"] Nov 02 03:17:47.734811 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:17:47.734781 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb040488f_08c5_4d4b_b3de_1f2bb279b747.slice/crio-78e755d449cdecb188c8543b4323d342806dbc5b1692e0e505c06ed7dba0bb48 WatchSource:0}: Error finding container 78e755d449cdecb188c8543b4323d342806dbc5b1692e0e505c06ed7dba0bb48: Status 404 returned error can't find the container with id 78e755d449cdecb188c8543b4323d342806dbc5b1692e0e505c06ed7dba0bb48 Nov 02 03:17:47.737223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.737184 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:17:47.737288 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.737273 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:17:47.737325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.737314 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:17:47.835457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.835421 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" event={"ID":"b040488f-08c5-4d4b-b3de-1f2bb279b747","Type":"ContainerStarted","Data":"2d1353031f16f9235cb0aca9e3b7dd7afaae95bf02f215fac0e058a2ef7c9835"} Nov 02 03:17:47.835618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.835477 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" event={"ID":"b040488f-08c5-4d4b-b3de-1f2bb279b747","Type":"ContainerStarted","Data":"78e755d449cdecb188c8543b4323d342806dbc5b1692e0e505c06ed7dba0bb48"} Nov 02 03:17:47.953144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:47.953089 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podStartSLOduration=1.953075615 podStartE2EDuration="1.953075615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:17:47.952514171 +0000 UTC m=+1809.948215133" watchObservedRunningTime="2026-11-02 03:17:47.953075615 +0000 UTC m=+1809.948776618" Nov 02 03:17:48.313147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:48.313114 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:48.314915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:48.314883 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 03:17:48.315032 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:48.314931 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 03:17:49.313837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:49.313804 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 03:17:49.314226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:49.313860 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 03:17:50.313275 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:50.313242 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 03:17:50.313442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:50.313302 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 03:17:51.313198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:51.313163 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 03:17:51.313704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:51.313216 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 03:17:52.313525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:52.313492 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 03:17:52.313934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:52.313546 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 03:17:53.313816 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:53.313782 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 03:17:53.314196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:53.313836 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 03:17:54.631334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:54.631302 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:54.631880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:54.631848 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:17:54.632567 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:17:54.632547 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:18:03.446565 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:18:03.446526 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:18:05.299378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.297395 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 03:18:05.336417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.330434 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:05.378519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.378472 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:05.460287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.459449 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:05.503487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.503441 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:05.550705 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.550586 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:05.568647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.568593 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:05.659377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:05.633827 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:06.306803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.306755 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 03:18:06.329838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.308149 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:18:06.376376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.334464 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 02 03:18:06.376376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.334889 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:18:06.448377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.379215 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:06.448377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.379609 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:18:06.494384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.494323 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Nov 02 03:18:06.495079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.495057 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:18:06.506503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.505445 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:06.506503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.505826 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:18:06.619433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.554829 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 02 03:18:06.620004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.585458 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:06.652376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.620656 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:18:06.652717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.652691 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:18:06.669375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.654533 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:06.710421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.697965 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:18:06.789384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:06.748445 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.166374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.165805 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.326610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.326555 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 03:18:07.374879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.337592 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.393381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.347546 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:18:07.393663 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.390003 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.411156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.411114 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.504656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.504556 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Nov 02 03:18:07.538644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.538584 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.637378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.636441 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.637378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.636785 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.703403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.699506 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:07.972491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:07.972443 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:08.378298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.378151 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:08.378298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.378242 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:08.442376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.403170 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:08.472378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.464757 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 03:18:08.472676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.472655 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:18:08.480540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.480503 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:18:08.513378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.511423 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:08.549516 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.549473 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:08.838433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:08.823907 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:18:09.460593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.460034 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:09.501557 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.499422 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:09.501557 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.501472 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:09.535394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.516134 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Nov 02 03:18:09.550381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.550250 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:09.611705 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.610761 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:18:09.788022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.787941 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:18:09.906908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:09.906863 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:18:10.219758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:10.219725 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 03:18:17.255826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.255794 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7"] Nov 02 03:18:17.259388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.259343 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.302371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.302323 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.302522 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.302499 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.302604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.302540 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.302661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.302634 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.302759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.302742 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsd97\" (UniqueName: \"kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.302859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.302835 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.362496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.362469 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-858236\"/\"distinct-2-w3-858236-kserve-self-signed-certs\"" Nov 02 03:18:17.365004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.364982 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7"] Nov 02 03:18:17.404299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404265 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsd97\" (UniqueName: \"kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404331 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404420 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404492 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404520 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404574 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404939 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.404962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.404960 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.405140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.405025 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.421081 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.421017 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.471544 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.471031 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.487728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.487684 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsd97\" (UniqueName: \"kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97\") pod \"distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.572105 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.572026 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:17.954238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:17.954209 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7"] Nov 02 03:18:17.955792 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:18:17.955765 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf75d7404_a123_4115_bd4d_3a1ff3bf77e2.slice/crio-af2bcf729259036bbff11f17614bf9dc8166a93227822e274ed686bb8efcdb6e WatchSource:0}: Error finding container af2bcf729259036bbff11f17614bf9dc8166a93227822e274ed686bb8efcdb6e: Status 404 returned error can't find the container with id af2bcf729259036bbff11f17614bf9dc8166a93227822e274ed686bb8efcdb6e Nov 02 03:18:18.839226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:18.839186 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" event={"ID":"f75d7404-a123-4115-bd4d-3a1ff3bf77e2","Type":"ContainerStarted","Data":"a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e"} Nov 02 03:18:18.839226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:18.839229 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" event={"ID":"f75d7404-a123-4115-bd4d-3a1ff3bf77e2","Type":"ContainerStarted","Data":"af2bcf729259036bbff11f17614bf9dc8166a93227822e274ed686bb8efcdb6e"} Nov 02 03:18:18.953939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:18.953844 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podStartSLOduration=1.9538315320000001 podStartE2EDuration="1.953831532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:18.952032675 +0000 UTC m=+1840.947733620" watchObservedRunningTime="2026-11-02 03:18:18.953831532 +0000 UTC m=+1840.949532475" Nov 02 03:18:19.572899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:19.572871 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:26.157949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.157914 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q"] Nov 02 03:18:26.161860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.161834 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.202908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.202875 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203068 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203192 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203313 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203237 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203344 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nvn\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203435 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203473 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203597 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.203703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.203686 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305147 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305198 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nvn\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305231 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305249 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305573 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305392 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305573 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305517 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305578 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305623 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305716 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305747 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305800 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.305990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.305961 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.306174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.306153 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.306303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.306283 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.307629 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.307606 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.307987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.307968 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.359817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.359789 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q"] Nov 02 03:18:26.363392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.363248 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.363534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.363514 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nvn\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-cn88q\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.476697 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.476610 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:26.972382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.967147 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q"] Nov 02 03:18:26.999610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.997893 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:18:26.999610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.997986 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:18:26.999610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:26.998029 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:18:27.573553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:27.573501 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:27.718174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:27.717779 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:27.936410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:27.934035 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" event={"ID":"8f7062c6-6519-43cd-8d34-13aa34a528a0","Type":"ContainerStarted","Data":"c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a"} Nov 02 03:18:27.936410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:27.934093 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" event={"ID":"8f7062c6-6519-43cd-8d34-13aa34a528a0","Type":"ContainerStarted","Data":"ccb0597838346431e7af7f9e7bc4673858414954a6c94addceeedd59a2104fde"} Nov 02 03:18:28.061202 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:28.061138 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podStartSLOduration=2.061119666 podStartE2EDuration="2.061119666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:28.0566888 +0000 UTC m=+1850.052389743" watchObservedRunningTime="2026-11-02 03:18:28.061119666 +0000 UTC m=+1850.056820610" Nov 02 03:18:28.145179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:28.145137 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:18:28.498376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:28.480418 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:28.498748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:28.498717 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:28.498938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:28.498911 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:29.480848 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:29.480803 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:29.481521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:29.481484 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:29.657485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:29.657437 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:29.798435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:29.794444 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.075547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.075442 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.405375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.403923 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 03:18:30.405375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.404262 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 03:18:30.405646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.405419 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.415410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.414906 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 02 03:18:30.477746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.477700 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:30.478016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.477989 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:30.515480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.515437 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.657542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.657435 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.658563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.658527 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.659096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.659074 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:18:30.892304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.891051 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:30.892304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.891461 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:18:30.979561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:30.979472 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 02 03:18:31.091584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.091528 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:18:31.095391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.091816 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:31.329207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.329109 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:18:31.329515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.329456 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:31.404891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.404830 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:31.405328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.405284 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 03:18:31.406392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.405594 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:18:31.406616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.406601 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:18:31.407547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.405874 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:31.410634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.408152 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:18:31.430608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.430574 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:18:31.430933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.430913 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:18:31.486387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.477765 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:31.486387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.477819 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:31.504397 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.503522 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:18:31.519457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.518646 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:18:31.526488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.525837 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:18:31.623705 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.623673 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:18:31.627407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.625789 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 03:18:31.627407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:31.626386 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:18:32.477777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:32.477741 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:32.477966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:32.477803 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:33.470417 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:18:33.470380 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:18:33.477493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:33.477472 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:33.477578 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:33.477521 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:34.483374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:34.482844 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:34.483374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:34.482916 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:35.477180 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:35.477144 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-cn88q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 03:18:35.477386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:35.477206 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 03:18:36.482058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:36.482028 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:36.482478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:36.482315 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:36.482896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:36.482875 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:18:41.314636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.314602 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj"] Nov 02 03:18:41.318670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.318644 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.365628 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.365598 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj"] Nov 02 03:18:41.411055 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411021 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411065 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411089 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411223 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411272 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411296 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411473 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411578 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411559 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcd6\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.411632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.411624 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512405 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512517 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512569 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcd6\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512636 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512687 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512730 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512764 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.512836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512796 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.513001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512834 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.513001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.512846 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.513121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.513035 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.513175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.513122 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.513299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.513277 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.513595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.513572 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.514928 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.514901 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.515504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.515486 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.568876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.568824 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcd6\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.568986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.568871 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-pvrnj\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:41.635858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:41.635832 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:42.059809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.059777 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj"] Nov 02 03:18:42.060388 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:18:42.060362 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbb30da_81d2_4166_b377_a49e88d82e4c.slice/crio-cb474e1615088e7ac6a7b8fcf9417257d9e5dca7f65759bbc99dd20c5b0da50d WatchSource:0}: Error finding container cb474e1615088e7ac6a7b8fcf9417257d9e5dca7f65759bbc99dd20c5b0da50d: Status 404 returned error can't find the container with id cb474e1615088e7ac6a7b8fcf9417257d9e5dca7f65759bbc99dd20c5b0da50d Nov 02 03:18:42.062829 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.062799 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:18:42.062904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.062861 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:18:42.062904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.062891 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:18:42.359952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.359914 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" event={"ID":"dcbb30da-81d2-4166-b377-a49e88d82e4c","Type":"ContainerStarted","Data":"3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93"} Nov 02 03:18:42.360397 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.359985 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" event={"ID":"dcbb30da-81d2-4166-b377-a49e88d82e4c","Type":"ContainerStarted","Data":"cb474e1615088e7ac6a7b8fcf9417257d9e5dca7f65759bbc99dd20c5b0da50d"} Nov 02 03:18:42.479324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.479270 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podStartSLOduration=1.479254778 podStartE2EDuration="1.479254778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:18:42.476371814 +0000 UTC m=+1864.472072756" watchObservedRunningTime="2026-11-02 03:18:42.479254778 +0000 UTC m=+1864.474955721" Nov 02 03:18:42.640545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:42.640437 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:43.660398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:43.650779 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 03:18:43.660398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:43.659440 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 03:18:44.648376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:44.640664 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 03:18:44.648376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:44.640739 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 03:18:45.644678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:45.644642 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 03:18:45.645938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:45.645905 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 03:18:46.649376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:46.642672 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 03:18:46.649376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:46.642747 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 03:18:47.636639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:47.636604 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 03:18:47.637458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:47.637417 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 03:18:48.636659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:48.636624 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 03:18:48.637125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:48.636675 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 03:18:49.641017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:49.640988 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:49.641473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:49.641266 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:18:49.642000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:18:49.641981 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:19:03.456933 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:19:03.456890 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:19:33.084050 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.084017 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb"] Nov 02 03:19:33.087970 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.087954 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.159516 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.159484 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d5dcca72\"/\"openshift-service-ca.crt\"" Nov 02 03:19:33.164057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.164027 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb"] Nov 02 03:19:33.180364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.180327 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.180491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.180370 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.180536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.180495 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.180536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.180522 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.180601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.180563 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgnlt\" (UniqueName: \"kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.180658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.180645 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.191637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.191611 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d5dcca72\"/\"kube-root-ca.crt\"" Nov 02 03:19:33.191815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.191723 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d5dcca72\"/\"e2e-lifecycle-model-d5dcca72-kserve-self-signed-certs\"" Nov 02 03:19:33.191815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.191759 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d5dcca72\"/\"default-dockercfg-mlg6w\"" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.282809 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.283011 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.283043 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.283118 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.283150 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.283209 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgnlt\" (UniqueName: \"kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.285765 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.286027 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.303776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.303681 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.304062 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.304040 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.327810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.327778 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.349568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.349443 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgnlt\" (UniqueName: \"kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt\") pod \"e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.399700 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.399666 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:33.464382 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:19:33.463675 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:19:33.830091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:33.830057 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb"] Nov 02 03:19:33.831467 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:19:33.831429 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda91a2691_b73c_42e7_b39f_8d2455a1539c.slice/crio-4ff6db0950a996208e2c1030ee17597252e535fb4f77df2b8fc9bdf49e29d2f8 WatchSource:0}: Error finding container 4ff6db0950a996208e2c1030ee17597252e535fb4f77df2b8fc9bdf49e29d2f8: Status 404 returned error can't find the container with id 4ff6db0950a996208e2c1030ee17597252e535fb4f77df2b8fc9bdf49e29d2f8 Nov 02 03:19:34.802915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:34.802876 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" event={"ID":"a91a2691-b73c-42e7-b39f-8d2455a1539c","Type":"ContainerStarted","Data":"aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f"} Nov 02 03:19:34.803316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:34.802928 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" event={"ID":"a91a2691-b73c-42e7-b39f-8d2455a1539c","Type":"ContainerStarted","Data":"4ff6db0950a996208e2c1030ee17597252e535fb4f77df2b8fc9bdf49e29d2f8"} Nov 02 03:19:34.921325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:34.921275 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" podStartSLOduration=1.921261253 podStartE2EDuration="1.921261253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:34.918181762 +0000 UTC m=+1916.913882718" watchObservedRunningTime="2026-11-02 03:19:34.921261253 +0000 UTC m=+1916.916962195" Nov 02 03:19:35.400091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:35.400056 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:43.400007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:43.399975 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:43.412059 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:43.412038 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:43.864852 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:43.864823 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:19:47.970380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:47.967443 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns"] Nov 02 03:19:47.981215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:47.981183 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.064292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.064264 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns"] Nov 02 03:19:48.120704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120675 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.120704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120704 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.120901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120774 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.120901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120813 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.120901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120838 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.120901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120854 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.121060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.120913 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.121060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.121020 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.121140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.121070 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7fb\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222623 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222659 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222713 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222783 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222824 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222851 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.222908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.222877 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223018 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223050 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223118 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7fb\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223367 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223301 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223422 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223742 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.223952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.223891 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.225290 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.225266 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.226147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.226124 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.280723 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.280695 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.281327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.281306 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7fb\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb\") pod \"e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.298415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.298395 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:48.722683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.722652 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns"] Nov 02 03:19:48.722992 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:19:48.722968 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd8f9bf_bff5_4e62_b07b_192caf116b4c.slice/crio-65b77b4f590b3eda01aac1e90202f9edf0c902b8a33dd81a526c4c0fce46b908 WatchSource:0}: Error finding container 65b77b4f590b3eda01aac1e90202f9edf0c902b8a33dd81a526c4c0fce46b908: Status 404 returned error can't find the container with id 65b77b4f590b3eda01aac1e90202f9edf0c902b8a33dd81a526c4c0fce46b908 Nov 02 03:19:48.725365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.725316 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:19:48.725455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.725421 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:19:48.725499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.725454 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:19:48.886451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.886345 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" event={"ID":"edd8f9bf-bff5-4e62-b07b-192caf116b4c","Type":"ContainerStarted","Data":"d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382"} Nov 02 03:19:48.886451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:48.886425 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" event={"ID":"edd8f9bf-bff5-4e62-b07b-192caf116b4c","Type":"ContainerStarted","Data":"65b77b4f590b3eda01aac1e90202f9edf0c902b8a33dd81a526c4c0fce46b908"} Nov 02 03:19:49.004829 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:49.004736 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podStartSLOduration=2.004721696 podStartE2EDuration="2.004721696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:19:49.000932036 +0000 UTC m=+1930.996633005" watchObservedRunningTime="2026-11-02 03:19:49.004721696 +0000 UTC m=+1931.000422638" Nov 02 03:19:49.299381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:49.299260 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:49.300903 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:49.300881 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:49.300967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:49.300933 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:50.299534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:50.299499 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:50.299940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:50.299554 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:51.312379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:51.305897 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:51.312379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:51.306004 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:52.443187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.443142 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:52.451373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.443868 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:52.571378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.552716 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 03:19:52.571378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.564663 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:52.616134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.616068 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:52.696548 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.696448 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 02 03:19:52.757376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.698465 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Nov 02 03:19:52.782859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.699235 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 03:19:52.783440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.709387 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 02 03:19:52.792431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.783125 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:52.856593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:52.856539 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" podUID="a91a2691-b73c-42e7-b39f-8d2455a1539c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:53.301377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.300722 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:53.301377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.300786 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:53.573951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.573849 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:53.575185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.575159 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:19:53.665570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.665528 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:53.666475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.666451 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:19:53.788377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.762377 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:53.788377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.782393 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:19:53.788377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.783981 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 02 03:19:53.829428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:53.808511 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:19:54.016617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.016559 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:54.206436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.125479 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:54.382545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.369724 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:54.397533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.397487 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:54.535425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.534976 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:54.576379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.574768 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:19:54.695376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.681322 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:54.695376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.693779 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded" Nov 02 03:19:54.777374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.767236 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Nov 02 03:19:54.792538 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.791482 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:19:54.901684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.901655 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 03:19:54.903107 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.903087 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:19:54.906141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:54.906122 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 03:19:55.299076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:55.299047 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:55.299279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:55.299258 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:56.299665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:56.299633 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:56.300061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:56.299688 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:57.298984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:57.298890 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 03:19:57.298984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:57.298968 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 03:19:59.082290 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:59.082259 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:59.082816 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:59.082615 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:19:59.083463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:19:59.083434 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:20:03.447222 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:20:03.447184 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:20:06.046035 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.046001 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz"] Nov 02 03:20:06.050164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.050144 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.153988 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154032 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154058 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154176 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154209 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154267 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154474 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgjp\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154518 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.154681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.154656 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.156630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.156611 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz"] Nov 02 03:20:06.255639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255614 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.255789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255656 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.255846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255783 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.255846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255831 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.255947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255863 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.255947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255900 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256062 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.255980 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256094 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256099 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgjp\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256147 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256181 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256222 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256293 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.256747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.256728 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.258278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.258253 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.258447 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.258431 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.312203 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.312140 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.312384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.312341 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgjp\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.363805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.363779 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:06.786380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.786340 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz"] Nov 02 03:20:06.787057 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:20:06.787032 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dd4605_ccae_43a5_8c81_5bd4c5ab01af.slice/crio-4f239c98c16fa6c3df1339b336860dff687556e9c3bb087cc5602cd4b1f46b8e WatchSource:0}: Error finding container 4f239c98c16fa6c3df1339b336860dff687556e9c3bb087cc5602cd4b1f46b8e: Status 404 returned error can't find the container with id 4f239c98c16fa6c3df1339b336860dff687556e9c3bb087cc5602cd4b1f46b8e Nov 02 03:20:06.789408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.789374 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:06.789490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.789440 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:06.789490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:06.789468 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:07.241178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:07.241140 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" event={"ID":"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af","Type":"ContainerStarted","Data":"dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5"} Nov 02 03:20:07.241630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:07.241191 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" event={"ID":"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af","Type":"ContainerStarted","Data":"4f239c98c16fa6c3df1339b336860dff687556e9c3bb087cc5602cd4b1f46b8e"} Nov 02 03:20:07.358048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:07.357997 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podStartSLOduration=1.357982599 podStartE2EDuration="1.357982599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:07.354878078 +0000 UTC m=+1949.350579023" watchObservedRunningTime="2026-11-02 03:20:07.357982599 +0000 UTC m=+1949.353683541" Nov 02 03:20:07.364642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:07.364615 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:07.366231 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:07.366208 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:07.366345 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:07.366251 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:08.364844 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:08.364801 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:08.365270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:08.364856 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:09.364527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:09.364490 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:09.364708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:09.364548 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:10.364290 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:10.364256 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:10.364708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:10.364310 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:11.364794 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:11.364757 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:11.365272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:11.364836 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:12.364798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:12.364762 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:12.365205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:12.364812 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:13.364458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:13.364418 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:13.364670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:13.364481 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:14.364400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:14.364319 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 02 03:20:14.365085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:14.365046 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 02 03:20:15.135490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.135445 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:20:15.135880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.135847 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.506432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.505437 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.506928 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.506584 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.663391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.655456 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.663391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.655859 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 03:20:15.663391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.656146 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.691483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.691438 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.795427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.794447 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.912693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.904636 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:15.950376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:15.944478 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.144374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.142448 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:20:16.144374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.142533 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.322370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.321959 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:20:16.322370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.322018 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.394541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.394429 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:20:16.394805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.394763 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.517661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.517606 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 03:20:16.519226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.519204 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:20:16.637835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.637724 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:20:16.637835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.637778 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.657919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.657547 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.658576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.658122 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:20:16.887385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.816875 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 02 03:20:16.887385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.816960 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Nov 02 03:20:16.887385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.841591 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:20:16.953007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.952864 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:16.953467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.953443 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:20:16.971291 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:16.971268 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 03:20:17.124375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.110141 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 03:20:17.171922 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.171883 4868 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:17.213019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.212944 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 03:20:17.231117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.231096 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 03:20:17.437785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.437749 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:20:17.438071 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.438030 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:20:17.627107 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.627080 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:17.627507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.627383 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:17.628172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:17.628154 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:20:21.169097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.169065 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm"] Nov 02 03:20:21.174068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.174048 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238136 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238203 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238308 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238382 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr66\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238436 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238500 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238534 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238621 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.238671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.238653 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.279922 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.279891 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm"] Nov 02 03:20:21.340086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340055 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340099 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr66\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340118 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340164 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340196 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340273 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340315 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340380 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340442 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340521 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340575 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340868 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340909 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.340993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.340974 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.342779 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.342757 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.343218 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.343200 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.396380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.396341 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.396526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.396511 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr66\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.489735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.489661 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:21.916457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.916434 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm"] Nov 02 03:20:21.917602 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:20:21.917578 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c0d05e_9735_4e13_aa50_8d4b23671367.slice/crio-d10d67514c70e08c6d82d4a10f87b5378a249a57eb33116458f0428025c2b4aa WatchSource:0}: Error finding container d10d67514c70e08c6d82d4a10f87b5378a249a57eb33116458f0428025c2b4aa: Status 404 returned error can't find the container with id d10d67514c70e08c6d82d4a10f87b5378a249a57eb33116458f0428025c2b4aa Nov 02 03:20:21.920145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.920113 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:21.920216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.920180 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:21.920216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:21.920210 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:22.422045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:22.421917 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" event={"ID":"f9c0d05e-9735-4e13-aa50-8d4b23671367","Type":"ContainerStarted","Data":"a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641"} Nov 02 03:20:22.423944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:22.423099 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" event={"ID":"f9c0d05e-9735-4e13-aa50-8d4b23671367","Type":"ContainerStarted","Data":"d10d67514c70e08c6d82d4a10f87b5378a249a57eb33116458f0428025c2b4aa"} Nov 02 03:20:22.490659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:22.490632 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:22.491893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:22.491868 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:22.492013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:22.491914 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:23.490992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:23.490961 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:23.491514 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:23.491017 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:24.490423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:24.490382 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:24.490591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:24.490454 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:25.490238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:25.490200 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:25.490743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:25.490256 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:26.490797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:26.490764 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:26.491192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:26.490825 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:27.490736 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:27.490706 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:27.490933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:27.490764 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:28.490415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:28.490377 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 03:20:28.490592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:28.490439 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 03:20:29.495541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:29.495510 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:29.495980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:29.495957 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:29.496583 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:29.496563 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:20:29.662316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:29.662259 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podStartSLOduration=8.662241307 podStartE2EDuration="8.662241307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:22.547650889 +0000 UTC m=+1964.543351833" watchObservedRunningTime="2026-11-02 03:20:29.662241307 +0000 UTC m=+1971.657942251" Nov 02 03:20:33.445788 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:20:33.445752 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:20:34.874150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:34.874108 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx"] Nov 02 03:20:34.883202 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:34.883175 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.046984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.046951 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b4a230-openshift-default-dockercfg-9cds8\"" Nov 02 03:20:35.063138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.063098 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.063583 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.063543 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.063805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.063764 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.064007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.063991 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.064166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.064142 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.064296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.064282 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.064455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.064437 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.064617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.064594 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxss\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.064780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.064751 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.093674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.092291 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx"] Nov 02 03:20:35.244416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.244284 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.244712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.244687 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.244861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.244845 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.246153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.246127 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.248461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.248437 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxss\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.248893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.248874 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.249477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.249457 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.250119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.250100 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.251389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.251371 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.251643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.249853 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.251744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.246569 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.251822 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.251249 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.251900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.245769 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.252236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.252214 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.254619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.254594 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.273485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.273428 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.316377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.312962 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxss\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.316377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.315821 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token\") pod \"e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:35.514535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:35.514175 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:36.005924 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:20:36.005888 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912a2210_3718_4dcb_b421_3eda921fcdee.slice/crio-71a641cf04f3595d048c29ec494d3365d9fc0586bb079ed9509b74cb933d7d7a WatchSource:0}: Error finding container 71a641cf04f3595d048c29ec494d3365d9fc0586bb079ed9509b74cb933d7d7a: Status 404 returned error can't find the container with id 71a641cf04f3595d048c29ec494d3365d9fc0586bb079ed9509b74cb933d7d7a Nov 02 03:20:36.006683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.006652 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx"] Nov 02 03:20:36.008760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.008711 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:36.008881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.008785 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:36.008881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.008819 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:20:36.593028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.592993 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" event={"ID":"912a2210-3718-4dcb-b421-3eda921fcdee","Type":"ContainerStarted","Data":"651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b"} Nov 02 03:20:36.593200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.593051 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" event={"ID":"912a2210-3718-4dcb-b421-3eda921fcdee","Type":"ContainerStarted","Data":"71a641cf04f3595d048c29ec494d3365d9fc0586bb079ed9509b74cb933d7d7a"} Nov 02 03:20:36.774428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:36.773524 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podStartSLOduration=2.773501163 podStartE2EDuration="2.773501163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:36.707488375 +0000 UTC m=+1978.703189318" watchObservedRunningTime="2026-11-02 03:20:36.773501163 +0000 UTC m=+1978.769202108" Nov 02 03:20:37.516388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:37.516341 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:37.521551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:37.521528 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:37.599095 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:37.599074 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:37.599963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:37.599944 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:20:39.542025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.541996 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt"] Nov 02 03:20:39.546348 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.546329 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.628565 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.628535 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.628731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.628710 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.628793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.628766 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67w4\" (UniqueName: \"kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.628845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.628829 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.652022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.651995 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt"] Nov 02 03:20:39.670467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.670448 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0d26af-serving-cert\"" Nov 02 03:20:39.730105 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.730070 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.730246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.730138 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67w4\" (UniqueName: \"kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.730246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.730164 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.730246 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:20:39.730197 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-serving-cert: secret "maas-api-e2e-api-a-0d26af-serving-cert" not found Nov 02 03:20:39.730509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.730246 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.730509 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:20:39.730273 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls podName:d46087ea-96ce-4cf6-a236-b8732ecc76b8 nodeName:}" failed. No retries permitted until 2026-11-02 03:20:40.23025236 +0000 UTC m=+1982.225953283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls") pod "maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" (UID: "d46087ea-96ce-4cf6-a236-b8732ecc76b8") : secret "maas-api-e2e-api-a-0d26af-serving-cert" not found Nov 02 03:20:39.730872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.730849 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.732632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.732613 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:39.788205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:39.788175 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67w4\" (UniqueName: \"kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:40.239683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:40.239645 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:40.242161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:40.242133 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\") pod \"maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:40.459209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:40.459177 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:40.836519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:40.836484 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt"] Nov 02 03:20:40.837606 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:20:40.837582 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46087ea_96ce_4cf6_a236_b8732ecc76b8.slice/crio-ab25e5ff6b9398cd57330dc2a1a25115faef755fba2c02ed71f320e6a485ce56 WatchSource:0}: Error finding container ab25e5ff6b9398cd57330dc2a1a25115faef755fba2c02ed71f320e6a485ce56: Status 404 returned error can't find the container with id ab25e5ff6b9398cd57330dc2a1a25115faef755fba2c02ed71f320e6a485ce56 Nov 02 03:20:40.839146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:40.839128 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:20:41.623137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:41.623101 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" event={"ID":"d46087ea-96ce-4cf6-a236-b8732ecc76b8","Type":"ContainerStarted","Data":"ab25e5ff6b9398cd57330dc2a1a25115faef755fba2c02ed71f320e6a485ce56"} Nov 02 03:20:42.630939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:42.630902 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" event={"ID":"d46087ea-96ce-4cf6-a236-b8732ecc76b8","Type":"ContainerStarted","Data":"a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f"} Nov 02 03:20:42.631343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:42.631003 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:20:42.749116 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:42.749063 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" podStartSLOduration=2.9553416329999997 podStartE2EDuration="3.749047624s" totalImagesPullingTime="793.705991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:20:42.746315369 +0000 UTC m=+1984.742016336" watchObservedRunningTime="2026-11-02 03:20:42.749047624 +0000 UTC m=+1984.744748567" Nov 02 03:20:48.643503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:20:48.643474 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:21:03.445991 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:21:03.445948 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:21:33.445566 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:21:33.445524 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:21:33.844764 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.844676 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch"] Nov 02 03:21:33.850323 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.850291 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.884493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.884457 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.885341 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.885262 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.885611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.885592 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.885812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.885778 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkv65\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.886041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.886025 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.886227 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.886193 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.886563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.886490 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.886774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.886757 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:33.886990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:33.886975 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.023375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.016213 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c9d1fc-openshift-default-dockercfg-kxjcl\"" Nov 02 03:21:34.079725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.079685 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch"] Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.090967 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126265 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126384 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126479 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126552 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126577 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkv65\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126669 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126702 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.126762 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.128375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.092022 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.178308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.177878 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.203770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.203452 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.203770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.203716 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.214452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.214409 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.224677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.224612 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.254267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.252342 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.320379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.313786 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkv65\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.337090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.337014 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo\") pod \"e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.493752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.493651 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:34.935165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.935136 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch"] Nov 02 03:21:34.936030 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:21:34.936003 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a23d3ea_88c3_4d61_a544_d18c549e1aaf.slice/crio-bdf581d04d0f72dda2e44acd174f55bb33c4e2e8c074894e3b6ed0069ce054ee WatchSource:0}: Error finding container bdf581d04d0f72dda2e44acd174f55bb33c4e2e8c074894e3b6ed0069ce054ee: Status 404 returned error can't find the container with id bdf581d04d0f72dda2e44acd174f55bb33c4e2e8c074894e3b6ed0069ce054ee Nov 02 03:21:34.944466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.944428 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:21:34.944590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.944521 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:21:34.944590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:34.944565 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:21:35.382386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:35.382330 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" event={"ID":"7a23d3ea-88c3-4d61-a544-d18c549e1aaf","Type":"ContainerStarted","Data":"2b00e0135188ed33b77912b9adb5a0d6b6c067d068cd550338347281c64935be"} Nov 02 03:21:35.382531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:35.382432 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" event={"ID":"7a23d3ea-88c3-4d61-a544-d18c549e1aaf","Type":"ContainerStarted","Data":"bdf581d04d0f72dda2e44acd174f55bb33c4e2e8c074894e3b6ed0069ce054ee"} Nov 02 03:21:35.495558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:35.495529 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:35.501060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:35.501017 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podStartSLOduration=2.50100459 podStartE2EDuration="2.50100459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:35.497869023 +0000 UTC m=+2037.493569966" watchObservedRunningTime="2026-11-02 03:21:35.50100459 +0000 UTC m=+2037.496705532" Nov 02 03:21:36.002744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.002705 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn"] Nov 02 03:21:36.006792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.006771 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050546 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050644 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050673 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050710 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050736 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050785 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050923 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.050976 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsdb\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.054380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.051039 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.097374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.096413 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-018cfa-openshift-default-dockercfg-5tvc2\"" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.172926 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.172998 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsdb\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173049 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173135 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173189 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173217 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173260 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173292 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173342 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173369 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173721 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.173996 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.174406 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.182055 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.229820 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.258115 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn"] Nov 02 03:21:36.319886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.308727 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsdb\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.373032 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.372965 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.382003 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.381936 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token\") pod \"e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:36.524376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.524232 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:36.623086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:36.623052 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:37.099828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.099801 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn"] Nov 02 03:21:37.100753 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:21:37.100729 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c356f5e_2156_498e_a4e3_e3d3f87d1ba7.slice/crio-27f63dcc780ce5a926f7dc2692bab870d069adab1f702df673922486de6f7897 WatchSource:0}: Error finding container 27f63dcc780ce5a926f7dc2692bab870d069adab1f702df673922486de6f7897: Status 404 returned error can't find the container with id 27f63dcc780ce5a926f7dc2692bab870d069adab1f702df673922486de6f7897 Nov 02 03:21:37.104003 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.103970 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:21:37.104097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.104032 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:21:37.104097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.104061 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:21:37.447027 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.446990 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" event={"ID":"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7","Type":"ContainerStarted","Data":"2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a"} Nov 02 03:21:37.447233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.447054 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" event={"ID":"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7","Type":"ContainerStarted","Data":"27f63dcc780ce5a926f7dc2692bab870d069adab1f702df673922486de6f7897"} Nov 02 03:21:37.447429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.447407 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:37.448327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.448305 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:21:37.565090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.565036 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podStartSLOduration=2.565021561 podStartE2EDuration="2.565021561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:21:37.561995642 +0000 UTC m=+2039.557696585" watchObservedRunningTime="2026-11-02 03:21:37.565021561 +0000 UTC m=+2039.560722504" Nov 02 03:21:37.624819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:37.624791 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:38.630455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:38.630426 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:39.547910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:39.547882 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:21:39.556376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:21:39.549734 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:22:03.464711 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:22:03.464672 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:22:11.936012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:11.935955 4868 generic.go:356] "Generic (PLEG): container finished" podID="912a2210-3718-4dcb-b421-3eda921fcdee" containerID="651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b" exitCode=0 Nov 02 03:22:11.936385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:11.936025 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" event={"ID":"912a2210-3718-4dcb-b421-3eda921fcdee","Type":"ContainerDied","Data":"651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b"} Nov 02 03:22:11.936648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:11.936633 4868 scope.go:122] "RemoveContainer" containerID="651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b" Nov 02 03:22:11.938144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:11.938112 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:22:11.938239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:11.938196 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:22:11.938239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:11.938228 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:22:12.956376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:12.950014 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" event={"ID":"912a2210-3718-4dcb-b421-3eda921fcdee","Type":"ContainerStarted","Data":"9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11"} Nov 02 03:22:13.515199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:13.515166 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:22:13.515396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:13.515329 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 03:22:13.515445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:13.515409 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 03:22:14.515218 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:14.515180 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 03:22:14.515621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:14.515244 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 03:22:15.515233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:15.515198 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 03:22:15.515742 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:15.515269 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 03:22:16.515190 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:16.515157 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 03:22:16.515399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:16.515228 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 03:22:17.587374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:17.587063 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 03:22:17.587374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:17.587121 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 03:22:18.515180 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:18.515142 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 03:22:18.515379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:18.515207 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 03:22:19.519831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:19.519795 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:22:19.520261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:19.520193 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:22:19.521013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:19.520992 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:22:20.291828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.291787 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf"] Nov 02 03:22:20.296176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.296156 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.348961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.348927 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d6544b-serving-cert\"" Nov 02 03:22:20.353672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.353649 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf"] Nov 02 03:22:20.355722 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.355701 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.355800 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.355737 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vsf\" (UniqueName: \"kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.355992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.355973 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.356029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.356020 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.457191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.457159 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.457398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.457220 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.457398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.457299 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.457398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.457331 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vsf\" (UniqueName: \"kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.457582 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:22:20.457423 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-serving-cert: secret "maas-api-e2e-api-b-d6544b-serving-cert" not found Nov 02 03:22:20.457582 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:22:20.457530 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls podName:660ae859-15d0-42e5-acfb-6a164c51ff79 nodeName:}" failed. No retries permitted until 2026-11-02 03:22:20.957504257 +0000 UTC m=+2082.953205196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls") pod "maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" (UID: "660ae859-15d0-42e5-acfb-6a164c51ff79") : secret "maas-api-e2e-api-b-d6544b-serving-cert" not found Nov 02 03:22:20.458480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.458451 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.460669 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.460644 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.515296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.515272 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vsf\" (UniqueName: \"kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.995062 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.995027 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:20.997469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:20.997440 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\") pod \"maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:21.209316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:21.209284 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:21.588233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:21.588200 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf"] Nov 02 03:22:21.588678 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:22:21.588657 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660ae859_15d0_42e5_acfb_6a164c51ff79.slice/crio-edfc2a953edf54688646e7901a03d733fb5ca2935cd044ab33ded080aafbfa0b WatchSource:0}: Error finding container edfc2a953edf54688646e7901a03d733fb5ca2935cd044ab33ded080aafbfa0b: Status 404 returned error can't find the container with id edfc2a953edf54688646e7901a03d733fb5ca2935cd044ab33ded080aafbfa0b Nov 02 03:22:22.067834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:22.067791 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" event={"ID":"660ae859-15d0-42e5-acfb-6a164c51ff79","Type":"ContainerStarted","Data":"edfc2a953edf54688646e7901a03d733fb5ca2935cd044ab33ded080aafbfa0b"} Nov 02 03:22:23.076368 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:23.076316 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" event={"ID":"660ae859-15d0-42e5-acfb-6a164c51ff79","Type":"ContainerStarted","Data":"e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930"} Nov 02 03:22:23.076849 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:23.076397 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:23.191162 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:23.191112 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" podStartSLOduration=2.316812664 podStartE2EDuration="3.191099203s" totalImagesPullingTime="874.286539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:22:23.189550131 +0000 UTC m=+2085.185251074" watchObservedRunningTime="2026-11-02 03:22:23.191099203 +0000 UTC m=+2085.186800145" Nov 02 03:22:29.088236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:29.088207 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:22:33.536057 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:22:33.536002 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:22:38.336569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:38.336541 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:22:38.343832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:38.343802 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:22:59.390130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:59.390101 4868 generic.go:356] "Generic (PLEG): container finished" podID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerID="2b00e0135188ed33b77912b9adb5a0d6b6c067d068cd550338347281c64935be" exitCode=0 Nov 02 03:22:59.390502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:59.390171 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" event={"ID":"7a23d3ea-88c3-4d61-a544-d18c549e1aaf","Type":"ContainerDied","Data":"2b00e0135188ed33b77912b9adb5a0d6b6c067d068cd550338347281c64935be"} Nov 02 03:22:59.390580 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:59.390566 4868 scope.go:122] "RemoveContainer" containerID="2b00e0135188ed33b77912b9adb5a0d6b6c067d068cd550338347281c64935be" Nov 02 03:22:59.392077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:59.392050 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:22:59.392167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:59.392134 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:22:59.392207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:22:59.392168 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:23:00.399884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:00.399852 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" event={"ID":"7a23d3ea-88c3-4d61-a544-d18c549e1aaf","Type":"ContainerStarted","Data":"0ea29f115211327e3f12012a7f971e46e2d518e058b0acc564535b043791e5d8"} Nov 02 03:23:00.495540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:00.495508 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:23:00.495733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:00.495690 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 03:23:00.495809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:00.495738 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 03:23:01.498467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:01.495340 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 03:23:01.498467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:01.496941 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 03:23:02.495655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:02.495619 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 03:23:02.495828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:02.495682 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 03:23:03.469656 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:03.469607 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:23:03.495221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:03.495189 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 03:23:03.495365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:03.495248 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 03:23:04.495526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:04.495488 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 03:23:04.495944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:04.495552 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 03:23:05.495457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:05.495425 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 03:23:05.495632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:05.495474 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 03:23:06.499829 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:06.499798 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:23:06.500195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:06.500119 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:23:06.500889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:06.500870 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:23:23.479965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:23.479935 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9"] Nov 02 03:23:23.480654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:23.480628 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" containerID="cri-o://7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002" gracePeriod=30 Nov 02 03:23:28.783518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.783489 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:23:28.818694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.818632 4868 generic.go:356] "Generic (PLEG): container finished" podID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerID="2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a" exitCode=0 Nov 02 03:23:28.818910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.818890 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" event={"ID":"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7","Type":"ContainerDied","Data":"2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a"} Nov 02 03:23:28.819529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.819504 4868 scope.go:122] "RemoveContainer" containerID="2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a" Nov 02 03:23:28.822534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.822494 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:23:28.822797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.822772 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:23:28.822989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.822970 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:23:28.827128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.827101 4868 generic.go:356] "Generic (PLEG): container finished" podID="937c3c81-800d-4c11-9903-a8881d96162d" containerID="7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002" exitCode=0 Nov 02 03:23:28.827241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.827169 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" Nov 02 03:23:28.827241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.827179 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" event={"ID":"937c3c81-800d-4c11-9903-a8881d96162d","Type":"ContainerDied","Data":"7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002"} Nov 02 03:23:28.827241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.827218 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9" event={"ID":"937c3c81-800d-4c11-9903-a8881d96162d","Type":"ContainerDied","Data":"0df595036458c56888ffb1849fcfd20b6079b32ad5ba027aab1638f93dc9497e"} Nov 02 03:23:28.827241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.827242 4868 scope.go:122] "RemoveContainer" containerID="7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002" Nov 02 03:23:28.838136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.838116 4868 scope.go:122] "RemoveContainer" containerID="7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002" Nov 02 03:23:28.838447 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:28.838422 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002\": container with ID starting with 7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002 not found: ID does not exist" containerID="7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002" Nov 02 03:23:28.838512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.838457 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002"} err="failed to get container status \"7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002\": rpc error: code = NotFound desc = could not find container \"7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002\": container with ID starting with 7a1eb5512e03f90537505b2ac581ecdfd395b74cd0b5740da69309dcb5791002 not found: ID does not exist" Nov 02 03:23:28.903482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903460 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.903586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903523 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.903658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903628 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.903725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903693 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.903782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903722 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.903782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903747 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:28.903880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903825 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:28.903880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903833 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.903978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903948 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.904037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.903997 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:28.904037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.904015 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.904145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.904109 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:28.904145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.904117 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt\") pod \"937c3c81-800d-4c11-9903-a8881d96162d\" (UID: \"937c3c81-800d-4c11-9903-a8881d96162d\") " Nov 02 03:23:28.904314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.904296 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:23:28.905266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.905243 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:28.905399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.905271 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937c3c81-800d-4c11-9903-a8881d96162d-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:28.905399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.905285 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:28.905399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.905298 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:28.905399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.905328 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:28.906262 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.906241 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:28.906845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.906808 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:23:28.906957 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.906877 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "kube-api-access-stwwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:28.907155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:28.907135 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy" pod "937c3c81-800d-4c11-9903-a8881d96162d" (UID: "937c3c81-800d-4c11-9903-a8881d96162d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:29.006834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.006797 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stwwt\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-kube-api-access-stwwt\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:29.006834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.006836 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937c3c81-800d-4c11-9903-a8881d96162d-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:29.007030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.006852 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937c3c81-800d-4c11-9903-a8881d96162d-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:29.007030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.006867 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937c3c81-800d-4c11-9903-a8881d96162d-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:29.253247 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.251189 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9"] Nov 02 03:23:29.255154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.255128 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-w2ms9"] Nov 02 03:23:29.859987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:29.859946 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" event={"ID":"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7","Type":"ContainerStarted","Data":"7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6"} Nov 02 03:23:30.116952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:30.116879 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937c3c81-800d-4c11-9903-a8881d96162d" path="/var/lib/kubelet/pods/937c3c81-800d-4c11-9903-a8881d96162d/volumes" Nov 02 03:23:30.625560 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:30.625534 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:23:30.625760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:30.625733 4868 patch_prober.go:29] interesting pod/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 03:23:30.625850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:30.625788 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 03:23:31.625009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:31.624976 4868 patch_prober.go:29] interesting pod/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 03:23:31.625441 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:31.625029 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 03:23:32.625259 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:32.625225 4868 patch_prober.go:29] interesting pod/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 03:23:32.625679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:32.625277 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 03:23:33.635616 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:33.635533 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:23:33.636230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:33.636202 4868 patch_prober.go:29] interesting pod/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 03:23:33.636439 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:33.636415 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 03:23:34.625645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:34.625609 4868 patch_prober.go:29] interesting pod/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 03:23:34.625864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:34.625660 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 03:23:35.631083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:35.631049 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:23:35.631614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:35.631444 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:23:35.632200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:35.632177 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:23:40.226559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:40.226525 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w"] Nov 02 03:23:40.227076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:40.226827 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" containerID="cri-o://35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f" gracePeriod=30 Nov 02 03:23:40.277609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:40.277580 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98"] Nov 02 03:23:40.277906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:40.277862 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" containerID="cri-o://2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a" gracePeriod=30 Nov 02 03:23:43.494308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:43.494270 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q"] Nov 02 03:23:43.494713 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:43.494509 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" containerID="cri-o://c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a" gracePeriod=30 Nov 02 03:23:43.544152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:43.544120 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf"] Nov 02 03:23:43.544480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:43.544385 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" containerID="cri-o://35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799" gracePeriod=30 Nov 02 03:23:45.376211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.376182 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 02 03:23:45.376558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.376232 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 02 03:23:45.411196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.411161 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 02 03:23:45.411337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.411246 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 02 03:23:45.548223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.548191 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:23:45.598532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598506 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598664 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598547 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598664 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598586 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598702 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598802 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598823 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598856 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.598955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598916 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.598955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.598925 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.599108 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599094 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket\") pod \"01832d40-9375-4879-97b0-9e5723711232\" (UID: \"01832d40-9375-4879-97b0-9e5723711232\") " Nov 02 03:23:45.599252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599219 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:23:45.599252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599220 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.599384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599233 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.599593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599570 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.599867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599841 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01832d40-9375-4879-97b0-9e5723711232-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.599982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599869 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.599982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599885 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.599982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599898 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.599982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.599912 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.600969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.600937 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "kube-api-access-7hlhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:45.601066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.600982 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:45.601211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.601180 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.601379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.601337 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo" pod "01832d40-9375-4879-97b0-9e5723711232" (UID: "01832d40-9375-4879-97b0-9e5723711232"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:23:45.603838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.603823 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:23:45.700714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.700685 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.700859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.700722 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.700943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.700923 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.700986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.700955 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.700993 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701025 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701106 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701054 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701115 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701117 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.701259 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701153 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701259 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701181 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token\") pod \"63323a25-54f2-4c30-8154-662079ca62e4\" (UID: \"63323a25-54f2-4c30-8154-662079ca62e4\") " Nov 02 03:23:45.701392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701342 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.701585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701561 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.701664 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701483 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:23:45.701941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701924 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlhm\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-kube-api-access-7hlhm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701942 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01832d40-9375-4879-97b0-9e5723711232-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701952 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701961 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63323a25-54f2-4c30-8154-662079ca62e4-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701969 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701978 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701986 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.701995 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01832d40-9375-4879-97b0-9e5723711232-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.702034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.702004 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01832d40-9375-4879-97b0-9e5723711232-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.703023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.703000 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:45.703445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.703425 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:45.703691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.703674 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:23:45.703749 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.703695 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l" pod "63323a25-54f2-4c30-8154-662079ca62e4" (UID: "63323a25-54f2-4c30-8154-662079ca62e4"). InnerVolumeSpecName "kube-api-access-cxb8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:45.803646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.803540 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb8l\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-kube-api-access-cxb8l\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.804007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.803981 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63323a25-54f2-4c30-8154-662079ca62e4-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.804147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.804129 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63323a25-54f2-4c30-8154-662079ca62e4-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.804254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.804240 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63323a25-54f2-4c30-8154-662079ca62e4-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:45.960218 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.960186 4868 generic.go:356] "Generic (PLEG): container finished" podID="63323a25-54f2-4c30-8154-662079ca62e4" containerID="2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a" exitCode=0 Nov 02 03:23:45.960437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.960258 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" Nov 02 03:23:45.960437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.960266 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" event={"ID":"63323a25-54f2-4c30-8154-662079ca62e4","Type":"ContainerDied","Data":"2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a"} Nov 02 03:23:45.960437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.960389 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98" event={"ID":"63323a25-54f2-4c30-8154-662079ca62e4","Type":"ContainerDied","Data":"7535232a665e0f43a66362dacb9878119ed2733fee809874a583a28d408c0c86"} Nov 02 03:23:45.960437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.960422 4868 scope.go:122] "RemoveContainer" containerID="2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a" Nov 02 03:23:45.962220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.962183 4868 generic.go:356] "Generic (PLEG): container finished" podID="01832d40-9375-4879-97b0-9e5723711232" containerID="35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f" exitCode=0 Nov 02 03:23:45.962298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.962274 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" Nov 02 03:23:45.962426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.962404 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" event={"ID":"01832d40-9375-4879-97b0-9e5723711232","Type":"ContainerDied","Data":"35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f"} Nov 02 03:23:45.962495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.962454 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w" event={"ID":"01832d40-9375-4879-97b0-9e5723711232","Type":"ContainerDied","Data":"9f368a1a77a4d906453332779ba269e40def961dfc1c12c4455be0a43eb7b8df"} Nov 02 03:23:45.972778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.972757 4868 scope.go:122] "RemoveContainer" containerID="2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a" Nov 02 03:23:45.973075 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:45.973057 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a\": container with ID starting with 2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a not found: ID does not exist" containerID="2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a" Nov 02 03:23:45.973167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.973081 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a"} err="failed to get container status \"2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a\": rpc error: code = NotFound desc = could not find container \"2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a\": container with ID starting with 2ed453a0fad3ca206a63ce9e3619652d9ff8e72bebb25bb169802b465881df1a not found: ID does not exist" Nov 02 03:23:45.973167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.973100 4868 scope.go:122] "RemoveContainer" containerID="35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f" Nov 02 03:23:45.983442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.983417 4868 scope.go:122] "RemoveContainer" containerID="35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f" Nov 02 03:23:45.983736 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:45.983713 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f\": container with ID starting with 35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f not found: ID does not exist" containerID="35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f" Nov 02 03:23:45.983811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:45.983747 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f"} err="failed to get container status \"35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f\": rpc error: code = NotFound desc = could not find container \"35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f\": container with ID starting with 35cbc2ddc4c6ee94c77c559f64740bef6d550f6379bc65a68400bf3c1876272f not found: ID does not exist" Nov 02 03:23:46.126176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:46.126142 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98"] Nov 02 03:23:46.129155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:46.129130 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-xsk98"] Nov 02 03:23:46.235956 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:46.235921 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w"] Nov 02 03:23:46.239763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:46.239737 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-bnz6w"] Nov 02 03:23:48.119025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.118980 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01832d40-9375-4879-97b0-9e5723711232" path="/var/lib/kubelet/pods/01832d40-9375-4879-97b0-9e5723711232/volumes" Nov 02 03:23:48.119833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.119808 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63323a25-54f2-4c30-8154-662079ca62e4" path="/var/lib/kubelet/pods/63323a25-54f2-4c30-8154-662079ca62e4/volumes" Nov 02 03:23:48.833456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.833426 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:23:48.856819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.856799 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:23:48.979680 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.979649 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.979855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.979712 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.979855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.979773 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.979855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.979818 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.980023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.979920 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.980299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980272 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.980299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980293 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.980492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980301 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.980492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980420 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.980492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980452 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.980668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980524 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.980668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980603 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:23:48.980668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980634 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.980831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980775 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.980892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980871 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.980954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980890 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.980954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980918 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.981058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980960 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.981058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.980999 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.981158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981094 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.981158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981153 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.981261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981197 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy\") pod \"c151c523-e65a-4a2e-b75e-cfee2307c48a\" (UID: \"c151c523-e65a-4a2e-b75e-cfee2307c48a\") " Nov 02 03:23:48.981261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981243 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.981796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981313 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn\") pod \"8f7062c6-6519-43cd-8d34-13aa34a528a0\" (UID: \"8f7062c6-6519-43cd-8d34-13aa34a528a0\") " Nov 02 03:23:48.981796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981544 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.981796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981593 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:23:48.981796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.981716 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.982309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982281 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.982412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982382 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:23:48.982482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982437 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:23:48.982482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982458 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982477 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982492 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c151c523-e65a-4a2e-b75e-cfee2307c48a-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982525 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982539 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982552 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982567 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982580 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7062c6-6519-43cd-8d34-13aa34a528a0-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982593 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982607 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:48.982643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.982605 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:48.984016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.983991 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.984146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.984129 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "kube-api-access-v7nvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:48.984197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.984148 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy" pod "8f7062c6-6519-43cd-8d34-13aa34a528a0" (UID: "8f7062c6-6519-43cd-8d34-13aa34a528a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:23:48.984410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.984395 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:48.984594 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:48.984572 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r" pod "c151c523-e65a-4a2e-b75e-cfee2307c48a" (UID: "c151c523-e65a-4a2e-b75e-cfee2307c48a"). InnerVolumeSpecName "kube-api-access-fsm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084083 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nvn\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-kube-api-access-v7nvn\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084116 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084127 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084136 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084145 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7062c6-6519-43cd-8d34-13aa34a528a0-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084152 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084161 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsm4r\" (UniqueName: \"kubernetes.io/projected/c151c523-e65a-4a2e-b75e-cfee2307c48a-kube-api-access-fsm4r\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.084170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.084174 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c151c523-e65a-4a2e-b75e-cfee2307c48a-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:23:49.184730 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.184692 4868 generic.go:356] "Generic (PLEG): container finished" podID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerID="35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799" exitCode=0 Nov 02 03:23:49.185201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.184750 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" Nov 02 03:23:49.185201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.184787 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" event={"ID":"c151c523-e65a-4a2e-b75e-cfee2307c48a","Type":"ContainerDied","Data":"35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799"} Nov 02 03:23:49.185201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.184840 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf" event={"ID":"c151c523-e65a-4a2e-b75e-cfee2307c48a","Type":"ContainerDied","Data":"e8ab776d4055adf5998e9e0c1e502bf194ad3f002e80216fa63aafeea4d47978"} Nov 02 03:23:49.185201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.184870 4868 scope.go:122] "RemoveContainer" containerID="35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799" Nov 02 03:23:49.186269 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.186248 4868 generic.go:356] "Generic (PLEG): container finished" podID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerID="c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a" exitCode=0 Nov 02 03:23:49.186400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.186277 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" event={"ID":"8f7062c6-6519-43cd-8d34-13aa34a528a0","Type":"ContainerDied","Data":"c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a"} Nov 02 03:23:49.186400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.186297 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" Nov 02 03:23:49.186400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.186325 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q" event={"ID":"8f7062c6-6519-43cd-8d34-13aa34a528a0","Type":"ContainerDied","Data":"ccb0597838346431e7af7f9e7bc4673858414954a6c94addceeedd59a2104fde"} Nov 02 03:23:49.194758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.194734 4868 scope.go:122] "RemoveContainer" containerID="35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799" Nov 02 03:23:49.195064 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:49.195042 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799\": container with ID starting with 35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799 not found: ID does not exist" containerID="35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799" Nov 02 03:23:49.195138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.195070 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799"} err="failed to get container status \"35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799\": rpc error: code = NotFound desc = could not find container \"35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799\": container with ID starting with 35a36cf5e3e8166ee1d843f593fab1ca0d57d4754fca1d6052fc4d61d14a1799 not found: ID does not exist" Nov 02 03:23:49.195138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.195088 4868 scope.go:122] "RemoveContainer" containerID="c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a" Nov 02 03:23:49.204707 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.204685 4868 scope.go:122] "RemoveContainer" containerID="c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a" Nov 02 03:23:49.205061 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:23:49.205040 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a\": container with ID starting with c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a not found: ID does not exist" containerID="c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a" Nov 02 03:23:49.205147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.205070 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a"} err="failed to get container status \"c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a\": rpc error: code = NotFound desc = could not find container \"c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a\": container with ID starting with c02bb3a10074e2ce36d49517b7526ef4c6f6d42df11f3a87ba5fd9dd25bbc57a not found: ID does not exist" Nov 02 03:23:49.348393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.348293 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q"] Nov 02 03:23:49.352205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.352177 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-cn88q"] Nov 02 03:23:49.460724 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.460686 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf"] Nov 02 03:23:49.465337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:49.465307 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-6gsrf"] Nov 02 03:23:50.116076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:50.116043 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" path="/var/lib/kubelet/pods/8f7062c6-6519-43cd-8d34-13aa34a528a0/volumes" Nov 02 03:23:50.116542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:23:50.116519 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" path="/var/lib/kubelet/pods/c151c523-e65a-4a2e-b75e-cfee2307c48a/volumes" Nov 02 03:24:00.503542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.502887 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m"] Nov 02 03:24:00.504086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504063 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" Nov 02 03:24:00.504181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504088 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" Nov 02 03:24:00.504181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504111 4868 cpu_manager.go:414] "containerMap: removing container" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" Nov 02 03:24:00.504181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504119 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" Nov 02 03:24:00.504181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504165 4868 cpu_manager.go:414] "containerMap: removing container" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" Nov 02 03:24:00.504181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504174 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" Nov 02 03:24:00.504459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504208 4868 cpu_manager.go:414] "containerMap: removing container" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" Nov 02 03:24:00.504459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504216 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" Nov 02 03:24:00.504459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504233 4868 cpu_manager.go:414] "containerMap: removing container" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" Nov 02 03:24:00.504459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504240 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" Nov 02 03:24:00.504653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504490 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="937c3c81-800d-4c11-9903-a8881d96162d" containerName="istio-proxy" Nov 02 03:24:00.504653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504512 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="63323a25-54f2-4c30-8154-662079ca62e4" containerName="istio-proxy" Nov 02 03:24:00.504653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504528 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c151c523-e65a-4a2e-b75e-cfee2307c48a" containerName="istio-proxy" Nov 02 03:24:00.504653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504544 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="01832d40-9375-4879-97b0-9e5723711232" containerName="istio-proxy" Nov 02 03:24:00.504653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.504555 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7062c6-6519-43cd-8d34-13aa34a528a0" containerName="istio-proxy" Nov 02 03:24:00.516427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.516223 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.547384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.539207 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m"] Nov 02 03:24:00.641375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.627735 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c170e9\"" Nov 02 03:24:00.672425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.636489 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c170e9-dockercfg-wbvrv\"" Nov 02 03:24:00.672425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.649331 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.672425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.666795 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9vw\" (UniqueName: \"kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw\") pod \"payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.772507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.772415 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9vw\" (UniqueName: \"kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw\") pod \"payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.772670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.772609 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.773608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.773578 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.829862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.829833 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9vw\" (UniqueName: \"kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw\") pod \"payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:00.850616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:00.850589 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:01.221908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:01.221875 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m"] Nov 02 03:24:01.225497 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:24:01.225470 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65e43f3_b3eb_4279_bbb8_d6a64021b15d.slice/crio-41911a9d0ab710e48f8381e895d5123cd0dc8248bc680a487545096f529f69ae WatchSource:0}: Error finding container 41911a9d0ab710e48f8381e895d5123cd0dc8248bc680a487545096f529f69ae: Status 404 returned error can't find the container with id 41911a9d0ab710e48f8381e895d5123cd0dc8248bc680a487545096f529f69ae Nov 02 03:24:01.376402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:01.376365 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" event={"ID":"e65e43f3-b3eb-4279-bbb8-d6a64021b15d","Type":"ContainerStarted","Data":"41911a9d0ab710e48f8381e895d5123cd0dc8248bc680a487545096f529f69ae"} Nov 02 03:24:03.447671 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:24:03.447632 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:24:04.398652 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:04.398622 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" event={"ID":"e65e43f3-b3eb-4279-bbb8-d6a64021b15d","Type":"ContainerStarted","Data":"c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f"} Nov 02 03:24:04.398830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:04.398683 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:04.512888 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:04.512775 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" podStartSLOduration=1.805917858 podStartE2EDuration="4.512756093s" totalImagesPullingTime="2.706838235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:24:04.511670223 +0000 UTC m=+2186.507371162" watchObservedRunningTime="2026-11-02 03:24:04.512756093 +0000 UTC m=+2186.508457039" Nov 02 03:24:11.008209 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.008174 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss"] Nov 02 03:24:11.012513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.012492 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.068655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.068626 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss"] Nov 02 03:24:11.114186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.114161 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d5dcca72-serving-cert\"" Nov 02 03:24:11.141730 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.141705 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.141871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.141766 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhmm\" (UniqueName: \"kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.141940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.141920 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.141987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.141950 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.243030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.242997 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.243185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.243129 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.243185 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:24:11.243145 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-serving-cert: secret "maas-api-e2e-mt-d5dcca72-serving-cert" not found Nov 02 03:24:11.243185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.243173 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhmm\" (UniqueName: \"kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.243303 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:24:11.243196 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls podName:4f2e68b2-326a-4684-8445-7356e54c6dec nodeName:}" failed. No retries permitted until 2026-11-02 03:24:11.743183626 +0000 UTC m=+2193.738884547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls") pod "maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" (UID: "4f2e68b2-326a-4684-8445-7356e54c6dec") : secret "maas-api-e2e-mt-d5dcca72-serving-cert" not found Nov 02 03:24:11.243362 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.243311 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.243801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.243780 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.245637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.245615 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.325778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.312099 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhmm\" (UniqueName: \"kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.752334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.752298 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.754694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.754675 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\") pod \"maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:11.925404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:11.925380 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:12.297745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:12.297714 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss"] Nov 02 03:24:12.299381 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:24:12.299329 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2e68b2_326a_4684_8445_7356e54c6dec.slice/crio-fcd14aeb21632ec6034f229273fc71ee3839a355080facaf249ed59c9420a6f4 WatchSource:0}: Error finding container fcd14aeb21632ec6034f229273fc71ee3839a355080facaf249ed59c9420a6f4: Status 404 returned error can't find the container with id fcd14aeb21632ec6034f229273fc71ee3839a355080facaf249ed59c9420a6f4 Nov 02 03:24:12.481314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:12.481282 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" event={"ID":"4f2e68b2-326a-4684-8445-7356e54c6dec","Type":"ContainerStarted","Data":"fcd14aeb21632ec6034f229273fc71ee3839a355080facaf249ed59c9420a6f4"} Nov 02 03:24:13.488391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:13.488340 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" event={"ID":"4f2e68b2-326a-4684-8445-7356e54c6dec","Type":"ContainerStarted","Data":"4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66"} Nov 02 03:24:13.488782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:13.488424 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:13.603600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:13.603547 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" podStartSLOduration=2.812065074 podStartE2EDuration="3.603534425s" totalImagesPullingTime="791.469351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:24:13.601121087 +0000 UTC m=+2195.596822030" watchObservedRunningTime="2026-11-02 03:24:13.603534425 +0000 UTC m=+2195.599235367" Nov 02 03:24:15.410530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:15.410499 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:24:19.502139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:19.502067 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:24:33.609003 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:24:33.608967 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:24:51.387120 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.387086 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq"] Nov 02 03:24:51.394696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.394676 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.448064 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.448037 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq"] Nov 02 03:24:51.494335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.494306 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c170e9-serving-cert\"" Nov 02 03:24:51.522073 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.522043 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.522199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.522117 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.522365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.522324 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf876\" (UniqueName: \"kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.522443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.522422 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.623249 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.623219 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.623456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.623285 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.623456 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:24:51.623391 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-serving-cert: secret "maas-api-e2e-api-b-c170e9-serving-cert" not found Nov 02 03:24:51.623456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.623427 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf876\" (UniqueName: \"kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.623638 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:24:51.623460 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls podName:3cbdf351-b349-442a-8226-ebbb05d0bacd nodeName:}" failed. No retries permitted until 2026-11-02 03:24:52.123445003 +0000 UTC m=+2234.119145928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls") pod "maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" (UID: "3cbdf351-b349-442a-8226-ebbb05d0bacd") : secret "maas-api-e2e-api-b-c170e9-serving-cert" not found Nov 02 03:24:51.623638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.623495 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.623954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.623933 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.625869 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.625851 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:51.687886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:51.687811 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf876\" (UniqueName: \"kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:52.131112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:52.131083 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:52.133499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:52.133470 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\") pod \"maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:52.307671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:52.307645 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:52.681936 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:52.681906 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq"] Nov 02 03:24:52.683269 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:24:52.683241 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbdf351_b349_442a_8226_ebbb05d0bacd.slice/crio-ba8884a2815e3e72685599b574aafb6e3dcb82dce944ff2d26ffa8bfa810c545 WatchSource:0}: Error finding container ba8884a2815e3e72685599b574aafb6e3dcb82dce944ff2d26ffa8bfa810c545: Status 404 returned error can't find the container with id ba8884a2815e3e72685599b574aafb6e3dcb82dce944ff2d26ffa8bfa810c545 Nov 02 03:24:52.736347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:52.736316 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" event={"ID":"3cbdf351-b349-442a-8226-ebbb05d0bacd","Type":"ContainerStarted","Data":"ba8884a2815e3e72685599b574aafb6e3dcb82dce944ff2d26ffa8bfa810c545"} Nov 02 03:24:53.746149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:53.746106 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" event={"ID":"3cbdf351-b349-442a-8226-ebbb05d0bacd","Type":"ContainerStarted","Data":"2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d"} Nov 02 03:24:53.746621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:53.746348 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:24:53.861683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:53.861613 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" podStartSLOduration=2.027695582 podStartE2EDuration="2.861592117s" totalImagesPullingTime="833.896535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:24:53.858591723 +0000 UTC m=+2235.854292667" watchObservedRunningTime="2026-11-02 03:24:53.861592117 +0000 UTC m=+2235.857293061" Nov 02 03:24:59.758041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:24:59.758008 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:25:00.749393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.749348 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb"] Nov 02 03:25:00.753445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.753430 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.805450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.805424 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d5dcca72-openshift-default-dockercfg-glcl8\"" Nov 02 03:25:00.846839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.846814 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.846940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.846892 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.846988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.846951 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.847086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.847063 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.847138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.847119 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8zp\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.847256 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.847239 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.847296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.847283 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.847388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.847373 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.847430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.847406 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.860160 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.860136 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb"] Nov 02 03:25:00.949123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949085 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949322 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949166 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949322 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949211 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949322 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949284 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949389 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949417 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8zp\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949522 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949573 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949587 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949582 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949673 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949687 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.949995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.949838 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.950414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.950392 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.952041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.952019 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:00.952179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:00.952163 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:01.005617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.005556 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8zp\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:01.005900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.005877 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:01.067385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.067359 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:01.496981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.496951 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb"] Nov 02 03:25:01.497974 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:25:01.497948 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f23275a_45ae_4424_8aca_b9af430072bb.slice/crio-cca899210c5dd2a6c6b830c083ce4b9eb073dedeb8d39c003c7b243aa6f66314 WatchSource:0}: Error finding container cca899210c5dd2a6c6b830c083ce4b9eb073dedeb8d39c003c7b243aa6f66314: Status 404 returned error can't find the container with id cca899210c5dd2a6c6b830c083ce4b9eb073dedeb8d39c003c7b243aa6f66314 Nov 02 03:25:01.500446 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.500413 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:01.500504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.500479 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:01.500536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.500510 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:01.857149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.857112 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" event={"ID":"7f23275a-45ae-4424-8aca-b9af430072bb","Type":"ContainerStarted","Data":"fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992"} Nov 02 03:25:01.857604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.857174 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" event={"ID":"7f23275a-45ae-4424-8aca-b9af430072bb","Type":"ContainerStarted","Data":"cca899210c5dd2a6c6b830c083ce4b9eb073dedeb8d39c003c7b243aa6f66314"} Nov 02 03:25:01.973865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:01.973814 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podStartSLOduration=1.973799458 podStartE2EDuration="1.973799458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:01.972122637 +0000 UTC m=+2243.967823581" watchObservedRunningTime="2026-11-02 03:25:01.973799458 +0000 UTC m=+2243.969500396" Nov 02 03:25:02.068339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:02.068299 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:02.069987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:02.069959 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 03:25:02.070126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:02.070012 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 03:25:03.068020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:03.067978 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 03:25:03.068456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:03.068041 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 03:25:03.467768 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:03.467732 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:25:04.067963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:04.067929 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 03:25:04.068146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:04.067979 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 03:25:05.067933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:05.067899 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 03:25:05.068104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:05.067961 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 03:25:06.067661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:06.067629 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 03:25:06.068037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:06.067683 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 03:25:07.068296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:07.068258 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 03:25:07.068746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:07.068326 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 03:25:08.073526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:08.073493 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:08.074026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:08.073867 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:08.074577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:08.074553 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:25:15.864610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.864572 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54"] Nov 02 03:25:15.870287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.869826 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.912603 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.912572 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc"] Nov 02 03:25:15.916624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.916607 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.969938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.969906 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54"] Nov 02 03:25:15.981156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981117 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.981297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981175 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.981297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981274 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48lw\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.981461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981341 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.981461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981419 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9n6d\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.981461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981446 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.981631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981541 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.981631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981586 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.981631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981626 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.981798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981653 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.981798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981745 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.981909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981849 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.981965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981912 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.982024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.981968 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.982024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.982004 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.982131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.982028 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:15.982131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.982093 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:15.982131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:15.982121 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.020461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.020428 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc"] Nov 02 03:25:16.083488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.083442 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.084678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.084653 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9n6d\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.084894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.084856 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.085014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.084598 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.085014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.084926 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.085014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.084952 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.085014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.084989 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.085314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085015 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.085314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085066 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.085314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085161 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.085314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085216 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.085314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085264 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.085314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085303 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.085685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085334 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.087201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085451 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.087343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.087319 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.087433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085957 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.087433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085997 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.087433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.086197 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.087433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.086794 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.087433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.087137 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.087433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.086976 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.087722 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.085559 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.087814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.087793 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.087935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.087921 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.088939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.088919 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.089109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.089092 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.089261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.089237 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.089608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.089584 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48lw\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.095063 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.095012 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.097576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.097492 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.124549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.123473 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.163733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.163234 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.170204 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.170172 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48lw\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.172908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.172876 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9n6d\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.185857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.185825 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token\") pod \"e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.191463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.191438 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:16.229626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.229601 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:16.623891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.623865 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54"] Nov 02 03:25:16.624444 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:25:16.624416 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfc71d5_9cdd_4ec8_8314_c0924e0a997d.slice/crio-1b4cd8ab9fcb202094c4e17242b126c9721177efd1854146089ae10cbba490c4 WatchSource:0}: Error finding container 1b4cd8ab9fcb202094c4e17242b126c9721177efd1854146089ae10cbba490c4: Status 404 returned error can't find the container with id 1b4cd8ab9fcb202094c4e17242b126c9721177efd1854146089ae10cbba490c4 Nov 02 03:25:16.628457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.627887 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:16.628457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.627965 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:16.628457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.628007 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:16.628457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.628311 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc"] Nov 02 03:25:16.630240 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:25:16.630215 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf377d42f_1621_4215_8b24_f98dd241e830.slice/crio-f2659db82f046bed1b5a07e88d60ddd346a654411904169b4c8adc7c19a2bb32 WatchSource:0}: Error finding container f2659db82f046bed1b5a07e88d60ddd346a654411904169b4c8adc7c19a2bb32: Status 404 returned error can't find the container with id f2659db82f046bed1b5a07e88d60ddd346a654411904169b4c8adc7c19a2bb32 Nov 02 03:25:16.632636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.632607 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:16.632728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.632689 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:16.632728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.632715 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:25:16.957264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.957162 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" event={"ID":"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d","Type":"ContainerStarted","Data":"8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3"} Nov 02 03:25:16.957264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.957232 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" event={"ID":"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d","Type":"ContainerStarted","Data":"1b4cd8ab9fcb202094c4e17242b126c9721177efd1854146089ae10cbba490c4"} Nov 02 03:25:16.959979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.959943 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" event={"ID":"f377d42f-1621-4215-8b24-f98dd241e830","Type":"ContainerStarted","Data":"6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4"} Nov 02 03:25:16.960123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:16.959997 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" event={"ID":"f377d42f-1621-4215-8b24-f98dd241e830","Type":"ContainerStarted","Data":"f2659db82f046bed1b5a07e88d60ddd346a654411904169b4c8adc7c19a2bb32"} Nov 02 03:25:17.080028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.079964 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podStartSLOduration=2.079944504 podStartE2EDuration="2.079944504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:17.076050486 +0000 UTC m=+2259.071751426" watchObservedRunningTime="2026-11-02 03:25:17.079944504 +0000 UTC m=+2259.075645448" Nov 02 03:25:17.192395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.192367 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:17.194045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.194014 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:17.194203 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.194068 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:17.196201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.196160 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podStartSLOduration=2.196146503 podStartE2EDuration="2.196146503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:25:17.192808423 +0000 UTC m=+2259.188509365" watchObservedRunningTime="2026-11-02 03:25:17.196146503 +0000 UTC m=+2259.191847518" Nov 02 03:25:17.229948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.229894 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:17.231072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.231045 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:17.231167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:17.231092 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:18.192323 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:18.192289 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:18.192728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:18.192373 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:18.230022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:18.229996 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:18.230156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:18.230056 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:19.192943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:19.192908 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:19.193427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:19.192974 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:19.230555 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:19.230525 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:19.230702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:19.230571 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:20.192050 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:20.192015 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:20.192240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:20.192064 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:20.230619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:20.230588 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:20.230948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:20.230636 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:21.192011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:21.191980 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:21.192271 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:21.192246 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:21.230274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:21.230244 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:21.230429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:21.230289 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:22.192315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:22.192271 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:22.192898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:22.192328 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:22.231221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:22.230651 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:22.231221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:22.230709 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:23.192183 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:23.192143 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 03:25:23.192374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:23.192201 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 03:25:23.229993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:23.229954 4868 patch_prober.go:29] interesting pod/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 03:25:23.230197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:23.230013 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 03:25:24.242949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:24.242913 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:24.243857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:24.243833 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:24.244956 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:24.244929 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:25:24.251221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:24.251192 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:24.251565 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:24.251542 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:24.252193 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:24.252176 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:25:32.342514 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.342476 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt"] Nov 02 03:25:32.342884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.342681 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" podUID="d46087ea-96ce-4cf6-a236-b8732ecc76b8" containerName="maas-api" containerID="cri-o://a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f" gracePeriod=30 Nov 02 03:25:32.877074 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.876760 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:25:32.970958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.970919 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca\") pod \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " Nov 02 03:25:32.971142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.971010 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4\" (UniqueName: \"kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4\") pod \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " Nov 02 03:25:32.971142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.971090 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs\") pod \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " Nov 02 03:25:32.971241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.971219 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\") pod \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\" (UID: \"d46087ea-96ce-4cf6-a236-b8732ecc76b8\") " Nov 02 03:25:32.971392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.971369 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca" pod "d46087ea-96ce-4cf6-a236-b8732ecc76b8" (UID: "d46087ea-96ce-4cf6-a236-b8732ecc76b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:32.971841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.971814 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46087ea-96ce-4cf6-a236-b8732ecc76b8-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:32.974275 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.974246 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4" pod "d46087ea-96ce-4cf6-a236-b8732ecc76b8" (UID: "d46087ea-96ce-4cf6-a236-b8732ecc76b8"). InnerVolumeSpecName "kube-api-access-w67w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:32.977137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.977112 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs" pod "d46087ea-96ce-4cf6-a236-b8732ecc76b8" (UID: "d46087ea-96ce-4cf6-a236-b8732ecc76b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:25:32.977401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:32.977379 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls" pod "d46087ea-96ce-4cf6-a236-b8732ecc76b8" (UID: "d46087ea-96ce-4cf6-a236-b8732ecc76b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:25:33.072797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.072730 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:33.072797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.072757 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67w4\" (UniqueName: \"kubernetes.io/projected/d46087ea-96ce-4cf6-a236-b8732ecc76b8-kube-api-access-w67w4\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:33.072797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.072768 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46087ea-96ce-4cf6-a236-b8732ecc76b8-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:33.111790 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.111759 4868 generic.go:356] "Generic (PLEG): container finished" podID="d46087ea-96ce-4cf6-a236-b8732ecc76b8" containerID="a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f" exitCode=0 Nov 02 03:25:33.111937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.111803 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" Nov 02 03:25:33.111937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.111833 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" event={"ID":"d46087ea-96ce-4cf6-a236-b8732ecc76b8","Type":"ContainerDied","Data":"a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f"} Nov 02 03:25:33.111937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.111883 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt" event={"ID":"d46087ea-96ce-4cf6-a236-b8732ecc76b8","Type":"ContainerDied","Data":"ab25e5ff6b9398cd57330dc2a1a25115faef755fba2c02ed71f320e6a485ce56"} Nov 02 03:25:33.111937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.111913 4868 scope.go:122] "RemoveContainer" containerID="a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f" Nov 02 03:25:33.138752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.138732 4868 scope.go:122] "RemoveContainer" containerID="a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f" Nov 02 03:25:33.139083 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:33.139066 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f\": container with ID starting with a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f not found: ID does not exist" containerID="a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f" Nov 02 03:25:33.139154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.139099 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f"} err="failed to get container status \"a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f\": rpc error: code = NotFound desc = could not find container \"a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f\": container with ID starting with a33a813b496796a0106fb3e3f4c021b120317786d78d357ff4b8207f0ab7e02f not found: ID does not exist" Nov 02 03:25:33.275234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.275206 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt"] Nov 02 03:25:33.278000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:33.277981 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0d26af-6777459bf7-tsbbt"] Nov 02 03:25:33.445757 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:33.445713 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:25:34.116782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:34.116753 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46087ea-96ce-4cf6-a236-b8732ecc76b8" path="/var/lib/kubelet/pods/d46087ea-96ce-4cf6-a236-b8732ecc76b8/volumes" Nov 02 03:25:35.573487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.573453 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs"] Nov 02 03:25:35.573920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.573673 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" containerID="cri-o://e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd" gracePeriod=30 Nov 02 03:25:35.630608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.630573 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns"] Nov 02 03:25:35.631137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.631100 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" containerID="cri-o://d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382" gracePeriod=30 Nov 02 03:25:35.636938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.636910 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t"] Nov 02 03:25:35.637122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.637099 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" containerID="cri-o://1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42" gracePeriod=30 Nov 02 03:25:35.671517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.671496 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw"] Nov 02 03:25:35.671675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:35.671656 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" containerID="cri-o://2d1353031f16f9235cb0aca9e3b7dd7afaae95bf02f215fac0e058a2ef7c9835" gracePeriod=30 Nov 02 03:25:38.547727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:38.547700 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz"] Nov 02 03:25:38.548141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:38.547995 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" containerID="cri-o://dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5" gracePeriod=30 Nov 02 03:25:38.605383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:38.602920 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm"] Nov 02 03:25:38.605383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:38.603175 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" containerID="cri-o://a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641" gracePeriod=30 Nov 02 03:25:39.345256 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:39.345218 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:25:39.345444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:39.345276 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:25:39.633773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:39.633732 4868 patch_prober.go:29] interesting pod/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:25:39.634230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:39.633800 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:25:40.930789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.930763 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.971938 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972006 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972062 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972109 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972153 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972190 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972425 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972535 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972570 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy\") pod \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\" (UID: \"00c7540a-eda8-4b7b-8ac3-a46bbace2b72\") " Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972646 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.972834 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.973325 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.973369 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.973487 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.973556 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:40.984398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.973991 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:40.993488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.993457 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:40.993644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.993618 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:40.993764 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.993646 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "kube-api-access-6p222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:40.995666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:40.995641 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token" pod "00c7540a-eda8-4b7b-8ac3-a46bbace2b72" (UID: "00c7540a-eda8-4b7b-8ac3-a46bbace2b72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.035758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.035734 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:25:41.038971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.038948 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.075915 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076048 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076089 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076125 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076174 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076220 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076269 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076303 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076386 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076447 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076489 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076532 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076579 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data\") pod \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\" (UID: \"edd8f9bf-bff5-4e62-b07b-192caf116b4c\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076621 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076711 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076888 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076936 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.076995 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t\") pod \"2a974a34-3184-4677-9893-d32e4193d600\" (UID: \"2a974a34-3184-4677-9893-d32e4193d600\") " Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077770 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077800 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077817 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077832 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p222\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-kube-api-access-6p222\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077848 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077861 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.083376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.077874 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c7540a-eda8-4b7b-8ac3-a46bbace2b72-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.085407 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "kube-api-access-cc78t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.088309 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.088615 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.088821 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.088987 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.093395 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.093585 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.093784 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.093992 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.094640 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.094869 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.095605 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.095936 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.101792 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.101844 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk"] Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.102129 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" containerID="cri-o://8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434" gracePeriod=30 Nov 02 03:25:41.135655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.104870 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert" pod "2a974a34-3184-4677-9893-d32e4193d600" (UID: "2a974a34-3184-4677-9893-d32e4193d600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:41.136560 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.136489 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:41.143391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.143117 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.143391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.143132 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb" pod "edd8f9bf-bff5-4e62-b07b-192caf116b4c" (UID: "edd8f9bf-bff5-4e62-b07b-192caf116b4c"). InnerVolumeSpecName "kube-api-access-fj7fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.158153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.157045 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd"] Nov 02 03:25:41.158153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.157340 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" containerID="cri-o://4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d" gracePeriod=30 Nov 02 03:25:41.166976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.166587 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8"] Nov 02 03:25:41.166976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.166806 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" containerID="cri-o://3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97" gracePeriod=30 Nov 02 03:25:41.173692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.167434 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h"] Nov 02 03:25:41.173692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.167638 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" containerID="cri-o://8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f" gracePeriod=30 Nov 02 03:25:41.181390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181344 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a974a34-3184-4677-9893-d32e4193d600-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181397 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181412 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181427 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc78t\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-kube-api-access-cc78t\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181441 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181454 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181467 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181481 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181495 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj7fb\" (UniqueName: \"kubernetes.io/projected/edd8f9bf-bff5-4e62-b07b-192caf116b4c-kube-api-access-fj7fb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181509 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181522 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181536 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a974a34-3184-4677-9893-d32e4193d600-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181551 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181564 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181576 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181590 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a974a34-3184-4677-9893-d32e4193d600-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181602 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd8f9bf-bff5-4e62-b07b-192caf116b4c-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.181833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.181616 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a974a34-3184-4677-9893-d32e4193d600-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.182685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.182661 4868 generic.go:356] "Generic (PLEG): container finished" podID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerID="d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382" exitCode=0 Nov 02 03:25:41.182779 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.182713 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" Nov 02 03:25:41.182779 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.182748 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" event={"ID":"edd8f9bf-bff5-4e62-b07b-192caf116b4c","Type":"ContainerDied","Data":"d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382"} Nov 02 03:25:41.182857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.182809 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns" event={"ID":"edd8f9bf-bff5-4e62-b07b-192caf116b4c","Type":"ContainerDied","Data":"65b77b4f590b3eda01aac1e90202f9edf0c902b8a33dd81a526c4c0fce46b908"} Nov 02 03:25:41.182857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.182840 4868 scope.go:122] "RemoveContainer" containerID="d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382" Nov 02 03:25:41.185331 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.185314 4868 generic.go:356] "Generic (PLEG): container finished" podID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerID="2d1353031f16f9235cb0aca9e3b7dd7afaae95bf02f215fac0e058a2ef7c9835" exitCode=0 Nov 02 03:25:41.185646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.185627 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" event={"ID":"b040488f-08c5-4d4b-b3de-1f2bb279b747","Type":"ContainerDied","Data":"2d1353031f16f9235cb0aca9e3b7dd7afaae95bf02f215fac0e058a2ef7c9835"} Nov 02 03:25:41.186021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.185840 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:25:41.196782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.190339 4868 generic.go:356] "Generic (PLEG): container finished" podID="2a974a34-3184-4677-9893-d32e4193d600" containerID="1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42" exitCode=0 Nov 02 03:25:41.196782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.190477 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" event={"ID":"2a974a34-3184-4677-9893-d32e4193d600","Type":"ContainerDied","Data":"1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42"} Nov 02 03:25:41.196782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.190514 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" event={"ID":"2a974a34-3184-4677-9893-d32e4193d600","Type":"ContainerDied","Data":"c9d52039ff3b8156e2fb0ac985a4cbd1f6d4ee13108072997778a5912b2354b9"} Nov 02 03:25:41.196782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.190578 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t" Nov 02 03:25:41.202425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.202403 4868 generic.go:356] "Generic (PLEG): container finished" podID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerID="e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd" exitCode=0 Nov 02 03:25:41.202534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.202449 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" event={"ID":"00c7540a-eda8-4b7b-8ac3-a46bbace2b72","Type":"ContainerDied","Data":"e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd"} Nov 02 03:25:41.202534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.202486 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" event={"ID":"00c7540a-eda8-4b7b-8ac3-a46bbace2b72","Type":"ContainerDied","Data":"659dad4160f1f13881d60fa6a9be67e36d2e575bbfefaa64a75129a8c8b9b6c3"} Nov 02 03:25:41.202644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.202560 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs" Nov 02 03:25:41.217095 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.217079 4868 scope.go:122] "RemoveContainer" containerID="d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382" Nov 02 03:25:41.225022 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:41.218505 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382\": container with ID starting with d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382 not found: ID does not exist" containerID="d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382" Nov 02 03:25:41.225022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.218541 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382"} err="failed to get container status \"d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382\": rpc error: code = NotFound desc = could not find container \"d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382\": container with ID starting with d19146c8103b520ee97e801fff27c02c93d1967c36b152a32adc2a57b2f57382 not found: ID does not exist" Nov 02 03:25:41.225022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.218563 4868 scope.go:122] "RemoveContainer" containerID="1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42" Nov 02 03:25:41.237296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.237265 4868 scope.go:122] "RemoveContainer" containerID="1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42" Nov 02 03:25:41.237775 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:41.237730 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42\": container with ID starting with 1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42 not found: ID does not exist" containerID="1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42" Nov 02 03:25:41.237855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.237775 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42"} err="failed to get container status \"1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42\": rpc error: code = NotFound desc = could not find container \"1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42\": container with ID starting with 1f2e203926ec11881c6a48cb28538ca0a4e754618a6130259857be9bb64d6c42 not found: ID does not exist" Nov 02 03:25:41.237855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.237800 4868 scope.go:122] "RemoveContainer" containerID="e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd" Nov 02 03:25:41.248624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.248579 4868 scope.go:122] "RemoveContainer" containerID="126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf" Nov 02 03:25:41.268642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.260648 4868 scope.go:122] "RemoveContainer" containerID="e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd" Nov 02 03:25:41.268642 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:41.261126 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd\": container with ID starting with e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd not found: ID does not exist" containerID="e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd" Nov 02 03:25:41.268642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.261162 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd"} err="failed to get container status \"e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd\": rpc error: code = NotFound desc = could not find container \"e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd\": container with ID starting with e99ac59d776bcac1a25d43e8fe867ee0d38497a14bd387576e4fc7fb3af396dd not found: ID does not exist" Nov 02 03:25:41.268642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.261221 4868 scope.go:122] "RemoveContainer" containerID="126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf" Nov 02 03:25:41.268642 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:41.261869 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf\": container with ID starting with 126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf not found: ID does not exist" containerID="126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf" Nov 02 03:25:41.268642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.261895 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf"} err="failed to get container status \"126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf\": rpc error: code = NotFound desc = could not find container \"126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf\": container with ID starting with 126f3cc4f4c698d1c4db9d38163b0d239d8da5a22981d3255d0bb447d386eacf not found: ID does not exist" Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.286805 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.286896 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.286928 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.287047 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.287168 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.287207 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.287248 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.287288 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.291964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.287326 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket\") pod \"b040488f-08c5-4d4b-b3de-1f2bb279b747\" (UID: \"b040488f-08c5-4d4b-b3de-1f2bb279b747\") " Nov 02 03:25:41.319781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.308694 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.326627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.322465 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.326627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.322662 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.326627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.322915 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:41.335433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.329445 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.365026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.335655 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:41.365026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.338438 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "kube-api-access-vrb97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.365026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.349156 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:41.365026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.349310 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token" pod "b040488f-08c5-4d4b-b3de-1f2bb279b747" (UID: "b040488f-08c5-4d4b-b3de-1f2bb279b747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417384 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417419 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrb97\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-kube-api-access-vrb97\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417435 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417451 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417465 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417478 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417494 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b040488f-08c5-4d4b-b3de-1f2bb279b747-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417508 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.419825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.417519 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b040488f-08c5-4d4b-b3de-1f2bb279b747-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:41.773103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.773006 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns"] Nov 02 03:25:41.777519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.777489 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-vlcns"] Nov 02 03:25:41.835669 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.835632 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-6j5w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:25:41.835782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.835697 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:25:41.884962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.884932 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t"] Nov 02 03:25:41.889096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:41.889072 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-hxj9t"] Nov 02 03:25:42.119387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.119328 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a974a34-3184-4677-9893-d32e4193d600" path="/var/lib/kubelet/pods/2a974a34-3184-4677-9893-d32e4193d600/volumes" Nov 02 03:25:42.120017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.119998 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" path="/var/lib/kubelet/pods/edd8f9bf-bff5-4e62-b07b-192caf116b4c/volumes" Nov 02 03:25:42.212362 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.212335 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" Nov 02 03:25:42.212473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.212331 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw" event={"ID":"b040488f-08c5-4d4b-b3de-1f2bb279b747","Type":"ContainerDied","Data":"78e755d449cdecb188c8543b4323d342806dbc5b1692e0e505c06ed7dba0bb48"} Nov 02 03:25:42.212536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.212488 4868 scope.go:122] "RemoveContainer" containerID="2d1353031f16f9235cb0aca9e3b7dd7afaae95bf02f215fac0e058a2ef7c9835" Nov 02 03:25:42.231096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.231070 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs"] Nov 02 03:25:42.235089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.235067 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-rr4xs"] Nov 02 03:25:42.456739 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.456662 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw"] Nov 02 03:25:42.460224 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:42.460199 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ca6546-openshift-default-994bcc7f5-lzgvw"] Nov 02 03:25:43.880139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.880118 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:25:43.904586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.904562 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:25:43.948181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948153 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948188 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948213 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948249 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948386 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948448 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948452 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:43.948651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948596 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:43.948651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948627 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948675 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:43.948746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948679 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948782 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert\") pod \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\" (UID: \"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af\") " Nov 02 03:25:43.948901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.948861 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:43.949173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.949153 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:43.949587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.949563 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:43.949671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.949591 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:43.949671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.949607 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:43.949671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.949622 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:43.949671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.949636 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:43.950608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.950587 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:43.950741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.950720 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "kube-api-access-jsgjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:43.950859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.950842 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:43.950909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:43.950846 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo" pod "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" (UID: "c1dd4605-ccae-43a5-8c81-5bd4c5ab01af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:44.050215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050182 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050250 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050290 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050333 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050476 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050508 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050510 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:44.050607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050550 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.050607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050582 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.050670 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert\") pod \"f9c0d05e-9735-4e13-aa50-8d4b23671367\" (UID: \"f9c0d05e-9735-4e13-aa50-8d4b23671367\") " Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051077 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051246 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgjp\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-kube-api-access-jsgjp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051267 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051283 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051297 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051311 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.051324 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.053670 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.053753 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "kube-api-access-scr66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.056963 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.056983 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.057069 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.058846 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:44.063637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.059043 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token" pod "f9c0d05e-9735-4e13-aa50-8d4b23671367" (UID: "f9c0d05e-9735-4e13-aa50-8d4b23671367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:44.118770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.118732 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" path="/var/lib/kubelet/pods/00c7540a-eda8-4b7b-8ac3-a46bbace2b72/volumes" Nov 02 03:25:44.119330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.119308 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" path="/var/lib/kubelet/pods/b040488f-08c5-4d4b-b3de-1f2bb279b747/volumes" Nov 02 03:25:44.152619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152597 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.152619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152620 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.152771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152631 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.152771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152640 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scr66\" (UniqueName: \"kubernetes.io/projected/f9c0d05e-9735-4e13-aa50-8d4b23671367-kube-api-access-scr66\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.152771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152650 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.152771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152658 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c0d05e-9735-4e13-aa50-8d4b23671367-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.152771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.152666 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c0d05e-9735-4e13-aa50-8d4b23671367-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:44.228604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.228564 4868 generic.go:356] "Generic (PLEG): container finished" podID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerID="dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5" exitCode=0 Nov 02 03:25:44.228786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.228620 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" Nov 02 03:25:44.228786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.228649 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" event={"ID":"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af","Type":"ContainerDied","Data":"dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5"} Nov 02 03:25:44.228786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.228714 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz" event={"ID":"c1dd4605-ccae-43a5-8c81-5bd4c5ab01af","Type":"ContainerDied","Data":"4f239c98c16fa6c3df1339b336860dff687556e9c3bb087cc5602cd4b1f46b8e"} Nov 02 03:25:44.228786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.228749 4868 scope.go:122] "RemoveContainer" containerID="dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5" Nov 02 03:25:44.230746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.230721 4868 generic.go:356] "Generic (PLEG): container finished" podID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerID="a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641" exitCode=0 Nov 02 03:25:44.230890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.230798 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" event={"ID":"f9c0d05e-9735-4e13-aa50-8d4b23671367","Type":"ContainerDied","Data":"a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641"} Nov 02 03:25:44.230890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.230833 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" event={"ID":"f9c0d05e-9735-4e13-aa50-8d4b23671367","Type":"ContainerDied","Data":"d10d67514c70e08c6d82d4a10f87b5378a249a57eb33116458f0428025c2b4aa"} Nov 02 03:25:44.231033 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.230898 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm" Nov 02 03:25:44.242489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.242463 4868 scope.go:122] "RemoveContainer" containerID="dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5" Nov 02 03:25:44.242794 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:44.242769 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5\": container with ID starting with dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5 not found: ID does not exist" containerID="dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5" Nov 02 03:25:44.242877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.242807 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5"} err="failed to get container status \"dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5\": rpc error: code = NotFound desc = could not find container \"dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5\": container with ID starting with dc37e5fec13e243242a9007206e31eb206861b0d7a706428b395e59358f4bbd5 not found: ID does not exist" Nov 02 03:25:44.242877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.242833 4868 scope.go:122] "RemoveContainer" containerID="a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641" Nov 02 03:25:44.252246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.252225 4868 scope.go:122] "RemoveContainer" containerID="a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641" Nov 02 03:25:44.252610 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:44.252584 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641\": container with ID starting with a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641 not found: ID does not exist" containerID="a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641" Nov 02 03:25:44.252708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.252617 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641"} err="failed to get container status \"a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641\": rpc error: code = NotFound desc = could not find container \"a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641\": container with ID starting with a88f6924496fdd86e2ecb51ca5b25b1ffbe858e812d7314e9b24b851f4757641 not found: ID does not exist" Nov 02 03:25:44.403327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.398984 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz"] Nov 02 03:25:44.403327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.403228 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-ltbbz"] Nov 02 03:25:44.540554 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.539848 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm"] Nov 02 03:25:44.540554 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:44.539905 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-lxgxm"] Nov 02 03:25:45.051367 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:45.051316 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5552h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:25:45.051730 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:45.051412 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:25:45.262122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:45.262086 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cjfrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:25:45.262268 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:45.262139 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:25:45.610400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:45.609536 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-xfdrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:25:45.610400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:45.609642 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:25:46.115461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.115432 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" path="/var/lib/kubelet/pods/c1dd4605-ccae-43a5-8c81-5bd4c5ab01af/volumes" Nov 02 03:25:46.115913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.115899 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" path="/var/lib/kubelet/pods/f9c0d05e-9735-4e13-aa50-8d4b23671367/volumes" Nov 02 03:25:46.453846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.453811 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:25:46.488886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.488857 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:25:46.534238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.534218 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:25:46.537553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.537531 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:25:46.601905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.601882 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.601916 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.601935 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.601958 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.601987 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602054 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602099 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602203 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.602300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602218 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602242 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.602300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602269 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602510 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602300 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602510 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602339 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602510 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602403 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602510 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602436 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.602679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602525 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602527 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:46.602679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602555 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602626 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602650 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.602890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602678 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602740 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket\") pod \"a827d914-fcc0-4f38-b440-b633f0794700\" (UID: \"a827d914-fcc0-4f38-b440-b633f0794700\") " Nov 02 03:25:46.602890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602775 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data\") pod \"e524d49f-5df9-48a0-8218-ada7e5466962\" (UID: \"e524d49f-5df9-48a0-8218-ada7e5466962\") " Nov 02 03:25:46.602890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602822 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.602890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602834 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.602890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.602844 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603775 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603834 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603856 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603871 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603883 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e524d49f-5df9-48a0-8218-ada7e5466962-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603897 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603910 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603924 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.604091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.603937 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a827d914-fcc0-4f38-b440-b633f0794700-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.605943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.604708 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.605943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.605475 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "kube-api-access-mgk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.606086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.605989 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:46.606810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.606778 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.606926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.606810 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:46.606926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.606842 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "kube-api-access-2grxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.607047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.607027 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.607110 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.607070 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token" pod "a827d914-fcc0-4f38-b440-b633f0794700" (UID: "a827d914-fcc0-4f38-b440-b633f0794700"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.607329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.607289 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy" pod "e524d49f-5df9-48a0-8218-ada7e5466962" (UID: "e524d49f-5df9-48a0-8218-ada7e5466962"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.705294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705220 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.705294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705261 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.705294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705291 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705380 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705408 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705463 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705558 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705559 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705573 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705622 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.705708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705627 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705727 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705803 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705839 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705882 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705904 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705919 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.705955 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706003 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706032 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706068 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt\") pod \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\" (UID: \"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706108 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs\") pod \"175bc536-a8fa-45fa-ad57-6858060cf3d4\" (UID: \"175bc536-a8fa-45fa-ad57-6858060cf3d4\") " Nov 02 03:25:46.706141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706116 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.706792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706151 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:46.706792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706294 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.706792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706507 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.706792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706652 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:25:46.706998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706859 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.706998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706785 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2grxl\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-kube-api-access-2grxl\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.706998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706940 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.706998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706957 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.706998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706972 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.706998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.706997 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707012 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707026 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e524d49f-5df9-48a0-8218-ada7e5466962-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707041 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175bc536-a8fa-45fa-ad57-6858060cf3d4-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707055 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707069 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a827d914-fcc0-4f38-b440-b633f0794700-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707085 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a827d914-fcc0-4f38-b440-b633f0794700-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707098 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707113 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707127 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707142 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707158 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a827d914-fcc0-4f38-b440-b633f0794700-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707174 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e524d49f-5df9-48a0-8218-ada7e5466962-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707188 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgk68\" (UniqueName: \"kubernetes.io/projected/e524d49f-5df9-48a0-8218-ada7e5466962-kube-api-access-mgk68\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.707303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.707201 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.708279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.708246 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "kube-api-access-l9fg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.708279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.708257 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.708511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.708272 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:46.708671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.708653 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:25:46.708827 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.708811 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.709126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.709112 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:25:46.709540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.709518 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt" pod "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" (UID: "d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29"). InnerVolumeSpecName "kube-api-access-nv4wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.709938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.709920 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token" pod "175bc536-a8fa-45fa-ad57-6858060cf3d4" (UID: "175bc536-a8fa-45fa-ad57-6858060cf3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:25:46.808347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808317 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808346 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808391 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fg8\" (UniqueName: \"kubernetes.io/projected/175bc536-a8fa-45fa-ad57-6858060cf3d4-kube-api-access-l9fg8\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808405 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808418 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808432 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808444 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808458 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4wt\" (UniqueName: \"kubernetes.io/projected/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29-kube-api-access-nv4wt\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:46.808524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:46.808472 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175bc536-a8fa-45fa-ad57-6858060cf3d4-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:25:47.253774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.253743 4868 generic.go:356] "Generic (PLEG): container finished" podID="a827d914-fcc0-4f38-b440-b633f0794700" containerID="4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d" exitCode=0 Nov 02 03:25:47.254145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.253784 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" Nov 02 03:25:47.254145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.253812 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" event={"ID":"a827d914-fcc0-4f38-b440-b633f0794700","Type":"ContainerDied","Data":"4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d"} Nov 02 03:25:47.254145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.253864 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd" event={"ID":"a827d914-fcc0-4f38-b440-b633f0794700","Type":"ContainerDied","Data":"30ff594fbac61db1cffb76150378a8043774a48bf9dbf258535aa6ec426149f1"} Nov 02 03:25:47.254145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.253894 4868 scope.go:122] "RemoveContainer" containerID="4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d" Nov 02 03:25:47.255335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.255309 4868 generic.go:356] "Generic (PLEG): container finished" podID="e524d49f-5df9-48a0-8218-ada7e5466962" containerID="8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434" exitCode=0 Nov 02 03:25:47.255489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.255390 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" event={"ID":"e524d49f-5df9-48a0-8218-ada7e5466962","Type":"ContainerDied","Data":"8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434"} Nov 02 03:25:47.255489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.255400 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" Nov 02 03:25:47.255489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.255439 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk" event={"ID":"e524d49f-5df9-48a0-8218-ada7e5466962","Type":"ContainerDied","Data":"228f7ee799465849e3e278b25732874b54a024ddbf5aea2af46bbfc0f60b7e13"} Nov 02 03:25:47.256876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.256846 4868 generic.go:356] "Generic (PLEG): container finished" podID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerID="8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f" exitCode=0 Nov 02 03:25:47.256993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.256875 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" event={"ID":"175bc536-a8fa-45fa-ad57-6858060cf3d4","Type":"ContainerDied","Data":"8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f"} Nov 02 03:25:47.256993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.256891 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" Nov 02 03:25:47.256993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.256920 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h" event={"ID":"175bc536-a8fa-45fa-ad57-6858060cf3d4","Type":"ContainerDied","Data":"a3bed5ff71f12735f5ce3bd979423548dedeb507579d8b735551886638185021"} Nov 02 03:25:47.258583 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.258562 4868 generic.go:356] "Generic (PLEG): container finished" podID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerID="3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97" exitCode=0 Nov 02 03:25:47.258719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.258598 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" event={"ID":"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29","Type":"ContainerDied","Data":"3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97"} Nov 02 03:25:47.258719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.258608 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" Nov 02 03:25:47.258719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.258629 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8" event={"ID":"d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29","Type":"ContainerDied","Data":"cb3db43818c93fde2e40d1713eff45bd1042fa143d9e964d3af507305c847529"} Nov 02 03:25:47.266579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.266346 4868 scope.go:122] "RemoveContainer" containerID="4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d" Nov 02 03:25:47.266953 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:47.266929 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d\": container with ID starting with 4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d not found: ID does not exist" containerID="4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d" Nov 02 03:25:47.267021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.266964 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d"} err="failed to get container status \"4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d\": rpc error: code = NotFound desc = could not find container \"4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d\": container with ID starting with 4c2a241152dbd00b0622ddaa15c79a6ee1456f93f6ab5e9a15063c32d194923d not found: ID does not exist" Nov 02 03:25:47.267021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.266985 4868 scope.go:122] "RemoveContainer" containerID="8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434" Nov 02 03:25:47.277369 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.277334 4868 scope.go:122] "RemoveContainer" containerID="8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434" Nov 02 03:25:47.277615 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:47.277598 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434\": container with ID starting with 8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434 not found: ID does not exist" containerID="8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434" Nov 02 03:25:47.277665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.277622 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434"} err="failed to get container status \"8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434\": rpc error: code = NotFound desc = could not find container \"8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434\": container with ID starting with 8b2063bebf030b4759a00f80e0edeeeb5ae7a8a6ccf060ce0d183008b1e7f434 not found: ID does not exist" Nov 02 03:25:47.277665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.277639 4868 scope.go:122] "RemoveContainer" containerID="8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f" Nov 02 03:25:47.286602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.286580 4868 scope.go:122] "RemoveContainer" containerID="8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f" Nov 02 03:25:47.286845 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:47.286827 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f\": container with ID starting with 8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f not found: ID does not exist" containerID="8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f" Nov 02 03:25:47.286895 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.286852 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f"} err="failed to get container status \"8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f\": rpc error: code = NotFound desc = could not find container \"8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f\": container with ID starting with 8a7406d28213972b33d92f554f833ba92720cc1d9c068db9744c88bc78707c9f not found: ID does not exist" Nov 02 03:25:47.286895 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.286867 4868 scope.go:122] "RemoveContainer" containerID="3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97" Nov 02 03:25:47.294868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.294851 4868 scope.go:122] "RemoveContainer" containerID="3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97" Nov 02 03:25:47.295102 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:25:47.295084 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97\": container with ID starting with 3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97 not found: ID does not exist" containerID="3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97" Nov 02 03:25:47.295145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.295111 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97"} err="failed to get container status \"3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97\": rpc error: code = NotFound desc = could not find container \"3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97\": container with ID starting with 3e565c8504251192af760efa39ec1098f8ced9ad71a2183e74c9975b86228d97 not found: ID does not exist" Nov 02 03:25:47.416800 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.416769 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8"] Nov 02 03:25:47.465635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.465564 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-6j5w8"] Nov 02 03:25:47.530569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.530544 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd"] Nov 02 03:25:47.534250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.534224 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-xfdrd"] Nov 02 03:25:47.641579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.641551 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk"] Nov 02 03:25:47.643908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.643889 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cjfrk"] Nov 02 03:25:47.750176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.750118 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h"] Nov 02 03:25:47.752500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:47.752477 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5552h"] Nov 02 03:25:48.130430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:48.130396 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" path="/var/lib/kubelet/pods/175bc536-a8fa-45fa-ad57-6858060cf3d4/volumes" Nov 02 03:25:48.131148 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:48.131120 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a827d914-fcc0-4f38-b440-b633f0794700" path="/var/lib/kubelet/pods/a827d914-fcc0-4f38-b440-b633f0794700/volumes" Nov 02 03:25:48.131690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:48.131674 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" path="/var/lib/kubelet/pods/d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29/volumes" Nov 02 03:25:48.132489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:25:48.132471 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" path="/var/lib/kubelet/pods/e524d49f-5df9-48a0-8218-ada7e5466962/volumes" Nov 02 03:26:03.446048 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:26:03.446011 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:26:18.763004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:18.762967 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss"] Nov 02 03:26:18.763539 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:18.763170 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" podUID="4f2e68b2-326a-4684-8445-7356e54c6dec" containerName="maas-api" containerID="cri-o://4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66" gracePeriod=30 Nov 02 03:26:19.134899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.134873 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:26:19.239907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.236076 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca\") pod \"4f2e68b2-326a-4684-8445-7356e54c6dec\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " Nov 02 03:26:19.239907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.236175 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\") pod \"4f2e68b2-326a-4684-8445-7356e54c6dec\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " Nov 02 03:26:19.239907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.236702 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca" pod "4f2e68b2-326a-4684-8445-7356e54c6dec" (UID: "4f2e68b2-326a-4684-8445-7356e54c6dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:26:19.240416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.240388 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs\") pod \"4f2e68b2-326a-4684-8445-7356e54c6dec\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " Nov 02 03:26:19.240508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.240480 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm\" (UniqueName: \"kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm\") pod \"4f2e68b2-326a-4684-8445-7356e54c6dec\" (UID: \"4f2e68b2-326a-4684-8445-7356e54c6dec\") " Nov 02 03:26:19.241097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.241079 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2e68b2-326a-4684-8445-7356e54c6dec-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:19.250424 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.241578 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs" pod "4f2e68b2-326a-4684-8445-7356e54c6dec" (UID: "4f2e68b2-326a-4684-8445-7356e54c6dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:26:19.261550 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.259903 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls" pod "4f2e68b2-326a-4684-8445-7356e54c6dec" (UID: "4f2e68b2-326a-4684-8445-7356e54c6dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:26:19.265524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.265489 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm" pod "4f2e68b2-326a-4684-8445-7356e54c6dec" (UID: "4f2e68b2-326a-4684-8445-7356e54c6dec"). InnerVolumeSpecName "kube-api-access-hmhmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:26:19.342079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.342048 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:19.342079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.342075 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f2e68b2-326a-4684-8445-7356e54c6dec-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:19.342079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.342085 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhmm\" (UniqueName: \"kubernetes.io/projected/4f2e68b2-326a-4684-8445-7356e54c6dec-kube-api-access-hmhmm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:19.472165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.472077 4868 generic.go:356] "Generic (PLEG): container finished" podID="4f2e68b2-326a-4684-8445-7356e54c6dec" containerID="4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66" exitCode=0 Nov 02 03:26:19.472165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.472138 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" Nov 02 03:26:19.472449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.472159 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" event={"ID":"4f2e68b2-326a-4684-8445-7356e54c6dec","Type":"ContainerDied","Data":"4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66"} Nov 02 03:26:19.472449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.472230 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss" event={"ID":"4f2e68b2-326a-4684-8445-7356e54c6dec","Type":"ContainerDied","Data":"fcd14aeb21632ec6034f229273fc71ee3839a355080facaf249ed59c9420a6f4"} Nov 02 03:26:19.472449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.472262 4868 scope.go:122] "RemoveContainer" containerID="4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66" Nov 02 03:26:19.483578 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.483560 4868 scope.go:122] "RemoveContainer" containerID="4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66" Nov 02 03:26:19.483823 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:26:19.483806 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66\": container with ID starting with 4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66 not found: ID does not exist" containerID="4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66" Nov 02 03:26:19.483862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.483833 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66"} err="failed to get container status \"4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66\": rpc error: code = NotFound desc = could not find container \"4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66\": container with ID starting with 4cdc2df650292bec5c5edeab04b5371ef28cc873135c3e61856db0cb32608f66 not found: ID does not exist" Nov 02 03:26:19.637586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.637130 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss"] Nov 02 03:26:19.643507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:19.643481 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dcca72-565b874c9b-ht4ss"] Nov 02 03:26:20.115362 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:20.115330 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2e68b2-326a-4684-8445-7356e54c6dec" path="/var/lib/kubelet/pods/4f2e68b2-326a-4684-8445-7356e54c6dec/volumes" Nov 02 03:26:26.284646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:26.284609 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb"] Nov 02 03:26:33.465480 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:26:33.465442 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:26:33.584618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:33.584584 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx"] Nov 02 03:26:33.584811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:33.584788 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" containerID="cri-o://9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11" gracePeriod=30 Nov 02 03:26:34.522657 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:34.522622 4868 patch_prober.go:29] interesting pod/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:26:34.523034 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:34.522673 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:26:38.839437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.839412 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:26:38.918223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918198 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918259 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918305 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918322 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918395 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918420 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918442 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918487 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918486 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:38.918817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918596 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy\") pod \"912a2210-3718-4dcb-b421-3eda921fcdee\" (UID: \"912a2210-3718-4dcb-b421-3eda921fcdee\") " Nov 02 03:26:38.918817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918614 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:38.918817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918649 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:38.918989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918954 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:38.919049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.918970 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:26:38.919049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.919033 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:38.919156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.919053 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:38.919156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.919065 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:38.920739 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.920699 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "kube-api-access-hpxss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:26:38.920853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.920748 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:26:38.920853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.920788 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:38.920853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:38.920818 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token" pod "912a2210-3718-4dcb-b421-3eda921fcdee" (UID: "912a2210-3718-4dcb-b421-3eda921fcdee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:26:39.020237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.020166 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:39.020237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.020204 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912a2210-3718-4dcb-b421-3eda921fcdee-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:39.020237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.020219 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:39.020237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.020234 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912a2210-3718-4dcb-b421-3eda921fcdee-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:39.020472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.020248 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxss\" (UniqueName: \"kubernetes.io/projected/912a2210-3718-4dcb-b421-3eda921fcdee-kube-api-access-hpxss\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:39.020472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.020266 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912a2210-3718-4dcb-b421-3eda921fcdee-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:39.600624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.600588 4868 generic.go:356] "Generic (PLEG): container finished" podID="912a2210-3718-4dcb-b421-3eda921fcdee" containerID="9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11" exitCode=0 Nov 02 03:26:39.600774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.600643 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" Nov 02 03:26:39.600774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.600666 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" event={"ID":"912a2210-3718-4dcb-b421-3eda921fcdee","Type":"ContainerDied","Data":"9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11"} Nov 02 03:26:39.600774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.600721 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx" event={"ID":"912a2210-3718-4dcb-b421-3eda921fcdee","Type":"ContainerDied","Data":"71a641cf04f3595d048c29ec494d3365d9fc0586bb079ed9509b74cb933d7d7a"} Nov 02 03:26:39.600774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.600743 4868 scope.go:122] "RemoveContainer" containerID="9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11" Nov 02 03:26:39.611198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.611176 4868 scope.go:122] "RemoveContainer" containerID="651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b" Nov 02 03:26:39.623238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.623222 4868 scope.go:122] "RemoveContainer" containerID="9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11" Nov 02 03:26:39.623553 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:26:39.623524 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11\": container with ID starting with 9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11 not found: ID does not exist" containerID="9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11" Nov 02 03:26:39.623649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.623559 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11"} err="failed to get container status \"9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11\": rpc error: code = NotFound desc = could not find container \"9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11\": container with ID starting with 9456a1dea0ed5d877188edd8241df0a4f2f838b9aa95ec39499c06e19cf76b11 not found: ID does not exist" Nov 02 03:26:39.623649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.623579 4868 scope.go:122] "RemoveContainer" containerID="651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b" Nov 02 03:26:39.623851 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:26:39.623825 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b\": container with ID starting with 651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b not found: ID does not exist" containerID="651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b" Nov 02 03:26:39.623941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.623859 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b"} err="failed to get container status \"651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b\": rpc error: code = NotFound desc = could not find container \"651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b\": container with ID starting with 651971f5d15c822dc0481024ad34ed504e0fa3e1a456d8e422a63aa9785c3c5b not found: ID does not exist" Nov 02 03:26:39.813402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.813373 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx"] Nov 02 03:26:39.862491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:39.862469 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b4a230-openshift-default-559b6d97b5-54djx"] Nov 02 03:26:40.115169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:40.115096 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" path="/var/lib/kubelet/pods/912a2210-3718-4dcb-b421-3eda921fcdee/volumes" Nov 02 03:26:41.305085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.305046 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" podUID="a91a2691-b73c-42e7-b39f-8d2455a1539c" containerName="main" containerID="cri-o://aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f" gracePeriod=45 Nov 02 03:26:41.564391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.564320 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:26:41.613561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.613530 4868 generic.go:356] "Generic (PLEG): container finished" podID="a91a2691-b73c-42e7-b39f-8d2455a1539c" containerID="aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f" exitCode=0 Nov 02 03:26:41.613714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.613581 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" Nov 02 03:26:41.613714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.613618 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" event={"ID":"a91a2691-b73c-42e7-b39f-8d2455a1539c","Type":"ContainerDied","Data":"aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f"} Nov 02 03:26:41.613714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.613680 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb" event={"ID":"a91a2691-b73c-42e7-b39f-8d2455a1539c","Type":"ContainerDied","Data":"4ff6db0950a996208e2c1030ee17597252e535fb4f77df2b8fc9bdf49e29d2f8"} Nov 02 03:26:41.613714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.613706 4868 scope.go:122] "RemoveContainer" containerID="aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f" Nov 02 03:26:41.622486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.622465 4868 scope.go:122] "RemoveContainer" containerID="aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f" Nov 02 03:26:41.622746 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:26:41.622730 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f\": container with ID starting with aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f not found: ID does not exist" containerID="aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f" Nov 02 03:26:41.622797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.622753 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f"} err="failed to get container status \"aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f\": rpc error: code = NotFound desc = could not find container \"aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f\": container with ID starting with aac8e1cc15f802911efc83bd7acfff5d8ad4f9b1ab70ac97f6fc25d8a4e9305f not found: ID does not exist" Nov 02 03:26:41.646495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.646474 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm\") pod \"a91a2691-b73c-42e7-b39f-8d2455a1539c\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " Nov 02 03:26:41.646591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.646545 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs\") pod \"a91a2691-b73c-42e7-b39f-8d2455a1539c\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " Nov 02 03:26:41.646645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.646613 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home\") pod \"a91a2691-b73c-42e7-b39f-8d2455a1539c\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " Nov 02 03:26:41.646679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.646655 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache\") pod \"a91a2691-b73c-42e7-b39f-8d2455a1539c\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " Nov 02 03:26:41.646732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.646690 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir\") pod \"a91a2691-b73c-42e7-b39f-8d2455a1539c\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " Nov 02 03:26:41.646732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.646717 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt\" (UniqueName: \"kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt\") pod \"a91a2691-b73c-42e7-b39f-8d2455a1539c\" (UID: \"a91a2691-b73c-42e7-b39f-8d2455a1539c\") " Nov 02 03:26:41.647047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.647018 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache" pod "a91a2691-b73c-42e7-b39f-8d2455a1539c" (UID: "a91a2691-b73c-42e7-b39f-8d2455a1539c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:41.647047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.647042 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir" pod "a91a2691-b73c-42e7-b39f-8d2455a1539c" (UID: "a91a2691-b73c-42e7-b39f-8d2455a1539c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:41.647213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.647142 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:41.647213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.647155 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:41.647213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.647204 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home" pod "a91a2691-b73c-42e7-b39f-8d2455a1539c" (UID: "a91a2691-b73c-42e7-b39f-8d2455a1539c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:41.648878 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.648848 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm" pod "a91a2691-b73c-42e7-b39f-8d2455a1539c" (UID: "a91a2691-b73c-42e7-b39f-8d2455a1539c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:26:41.648972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.648914 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt" pod "a91a2691-b73c-42e7-b39f-8d2455a1539c" (UID: "a91a2691-b73c-42e7-b39f-8d2455a1539c"). InnerVolumeSpecName "kube-api-access-hgnlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:26:41.648972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.648924 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs" pod "a91a2691-b73c-42e7-b39f-8d2455a1539c" (UID: "a91a2691-b73c-42e7-b39f-8d2455a1539c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:26:41.748619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.748589 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91a2691-b73c-42e7-b39f-8d2455a1539c-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:41.748619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.748617 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:41.748619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.748627 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgnlt\" (UniqueName: \"kubernetes.io/projected/a91a2691-b73c-42e7-b39f-8d2455a1539c-kube-api-access-hgnlt\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:41.748838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:41.748636 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91a2691-b73c-42e7-b39f-8d2455a1539c-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:26:42.032629 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:42.032602 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb"] Nov 02 03:26:42.036144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:42.036120 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d5dcca72/e2e-lifecycle-model-d5dcca72-kserve-548b467f4f-9rzcb"] Nov 02 03:26:42.114651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:26:42.114624 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91a2691-b73c-42e7-b39f-8d2455a1539c" path="/var/lib/kubelet/pods/a91a2691-b73c-42e7-b39f-8d2455a1539c/volumes" Nov 02 03:27:03.447186 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:03.447147 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:27:05.661653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:05.661623 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf"] Nov 02 03:27:05.662094 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:05.661811 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" podUID="660ae859-15d0-42e5-acfb-6a164c51ff79" containerName="maas-api" containerID="cri-o://e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930" gracePeriod=30 Nov 02 03:27:05.970494 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:05.969564 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:27:06.127729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.125776 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf\" (UniqueName: \"kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf\") pod \"660ae859-15d0-42e5-acfb-6a164c51ff79\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " Nov 02 03:27:06.127729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.125850 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\") pod \"660ae859-15d0-42e5-acfb-6a164c51ff79\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " Nov 02 03:27:06.127729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.125926 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs\") pod \"660ae859-15d0-42e5-acfb-6a164c51ff79\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " Nov 02 03:27:06.127729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.126099 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca\") pod \"660ae859-15d0-42e5-acfb-6a164c51ff79\" (UID: \"660ae859-15d0-42e5-acfb-6a164c51ff79\") " Nov 02 03:27:06.133906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.133845 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca" pod "660ae859-15d0-42e5-acfb-6a164c51ff79" (UID: "660ae859-15d0-42e5-acfb-6a164c51ff79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:06.154777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.154714 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf" pod "660ae859-15d0-42e5-acfb-6a164c51ff79" (UID: "660ae859-15d0-42e5-acfb-6a164c51ff79"). InnerVolumeSpecName "kube-api-access-d7vsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:06.173473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.173444 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls" pod "660ae859-15d0-42e5-acfb-6a164c51ff79" (UID: "660ae859-15d0-42e5-acfb-6a164c51ff79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:27:06.174090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.174063 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs" pod "660ae859-15d0-42e5-acfb-6a164c51ff79" (UID: "660ae859-15d0-42e5-acfb-6a164c51ff79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:27:06.229425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.229286 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ae859-15d0-42e5-acfb-6a164c51ff79-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:06.229425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.229328 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vsf\" (UniqueName: \"kubernetes.io/projected/660ae859-15d0-42e5-acfb-6a164c51ff79-kube-api-access-d7vsf\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:06.229425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.229344 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:06.229425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.229376 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ae859-15d0-42e5-acfb-6a164c51ff79-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:06.765013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.764983 4868 generic.go:356] "Generic (PLEG): container finished" podID="660ae859-15d0-42e5-acfb-6a164c51ff79" containerID="e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930" exitCode=0 Nov 02 03:27:06.765477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.765024 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" Nov 02 03:27:06.765477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.765063 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" event={"ID":"660ae859-15d0-42e5-acfb-6a164c51ff79","Type":"ContainerDied","Data":"e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930"} Nov 02 03:27:06.765477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.765117 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf" event={"ID":"660ae859-15d0-42e5-acfb-6a164c51ff79","Type":"ContainerDied","Data":"edfc2a953edf54688646e7901a03d733fb5ca2935cd044ab33ded080aafbfa0b"} Nov 02 03:27:06.765477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.765162 4868 scope.go:122] "RemoveContainer" containerID="e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930" Nov 02 03:27:06.775374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.775332 4868 scope.go:122] "RemoveContainer" containerID="e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930" Nov 02 03:27:06.775660 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:06.775642 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930\": container with ID starting with e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930 not found: ID does not exist" containerID="e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930" Nov 02 03:27:06.775726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.775669 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930"} err="failed to get container status \"e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930\": rpc error: code = NotFound desc = could not find container \"e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930\": container with ID starting with e986cf649f51c3cf36185957fe52baf11a0bb1687876897e73cfd5065a5df930 not found: ID does not exist" Nov 02 03:27:06.928341 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.928307 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf"] Nov 02 03:27:06.976117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:06.976033 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6544b-57499f6fb8-9cksf"] Nov 02 03:27:08.115618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:08.115587 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660ae859-15d0-42e5-acfb-6a164c51ff79" path="/var/lib/kubelet/pods/660ae859-15d0-42e5-acfb-6a164c51ff79/volumes" Nov 02 03:27:09.829143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:09.829103 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54"] Nov 02 03:27:09.829647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:09.829411 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" containerID="cri-o://8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3" gracePeriod=30 Nov 02 03:27:09.846212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:09.846182 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc"] Nov 02 03:27:09.846410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:09.846390 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" containerID="cri-o://6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4" gracePeriod=30 Nov 02 03:27:09.870173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:09.870150 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb"] Nov 02 03:27:09.870325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:09.870309 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" containerID="cri-o://fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992" gracePeriod=30 Nov 02 03:27:11.099029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.098993 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb"] Nov 02 03:27:11.099670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099649 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099672 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099690 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4f2e68b2-326a-4684-8445-7356e54c6dec" containerName="maas-api" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099697 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2e68b2-326a-4684-8445-7356e54c6dec" containerName="maas-api" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099706 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d46087ea-96ce-4cf6-a236-b8732ecc76b8" containerName="maas-api" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099714 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46087ea-96ce-4cf6-a236-b8732ecc76b8" containerName="maas-api" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099739 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" Nov 02 03:27:11.099759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099746 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099762 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099770 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099796 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099803 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099814 4868 cpu_manager.go:414] "containerMap: removing container" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099822 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099839 4868 cpu_manager.go:414] "containerMap: removing container" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099847 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099859 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099866 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099874 4868 cpu_manager.go:414] "containerMap: removing container" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099883 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099902 4868 cpu_manager.go:414] "containerMap: removing container" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099909 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099920 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099928 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099942 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a91a2691-b73c-42e7-b39f-8d2455a1539c" containerName="main" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099949 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91a2691-b73c-42e7-b39f-8d2455a1539c" containerName="main" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099965 4868 cpu_manager.go:414] "containerMap: removing container" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099973 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099983 4868 cpu_manager.go:414] "containerMap: removing container" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.099991 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100006 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100013 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100028 4868 cpu_manager.go:414] "containerMap: removing container" podUID="660ae859-15d0-42e5-acfb-6a164c51ff79" containerName="maas-api" Nov 02 03:27:11.100111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100036 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660ae859-15d0-42e5-acfb-6a164c51ff79" containerName="maas-api" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100159 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100171 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e524d49f-5df9-48a0-8218-ada7e5466962" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100182 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91a2691-b73c-42e7-b39f-8d2455a1539c" containerName="main" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100192 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c7540a-eda8-4b7b-8ac3-a46bbace2b72" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100202 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="175bc536-a8fa-45fa-ad57-6858060cf3d4" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100212 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a974a34-3184-4677-9893-d32e4193d600" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100220 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="660ae859-15d0-42e5-acfb-6a164c51ff79" containerName="maas-api" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100228 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6db303b-b46b-4ad6-a2ce-d5e8d1a6ad29" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100237 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2e68b2-326a-4684-8445-7356e54c6dec" containerName="maas-api" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100248 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46087ea-96ce-4cf6-a236-b8732ecc76b8" containerName="maas-api" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100258 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a827d914-fcc0-4f38-b440-b633f0794700" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100267 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b040488f-08c5-4d4b-b3de-1f2bb279b747" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100280 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c0d05e-9735-4e13-aa50-8d4b23671367" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100288 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100298 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd8f9bf-bff5-4e62-b07b-192caf116b4c" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100308 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dd4605-ccae-43a5-8c81-5bd4c5ab01af" containerName="istio-proxy" Nov 02 03:27:11.101326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.100317 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="912a2210-3718-4dcb-b421-3eda921fcdee" containerName="istio-proxy" Nov 02 03:27:11.105365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.105329 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.118801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.118776 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.118944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.118818 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.118944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.118852 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.118944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.118939 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.119124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.118958 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.119124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.119065 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws5sj\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.119234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.119118 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.119234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.119183 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.119340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.119272 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.180074 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.180039 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb"] Nov 02 03:27:11.219960 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.219930 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws5sj\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.219970 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220010 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220058 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220107 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220147 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220198 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220220 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220240 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220598 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220573 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220806 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.220934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.220887 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.221188 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.221066 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.225136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.222026 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.225136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.223181 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.225136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.223646 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.260805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.260686 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6a5285-openshift-default-dockercfg-llcrh\"" Nov 02 03:27:11.303395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.303363 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.309133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.309069 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws5sj\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj\") pod \"e2e-api-a-6a5285-openshift-default-6596676675-lfxqb\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.420419 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.420388 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:11.899702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.899673 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb"] Nov 02 03:27:11.901832 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:11.901800 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cfc3b9_7a65_4bb0_8245_ae992cde49d0.slice/crio-07aef3ab26ab7fff295017a10a73196bd68223ef6ca4187f030874f0947161ad WatchSource:0}: Error finding container 07aef3ab26ab7fff295017a10a73196bd68223ef6ca4187f030874f0947161ad: Status 404 returned error can't find the container with id 07aef3ab26ab7fff295017a10a73196bd68223ef6ca4187f030874f0947161ad Nov 02 03:27:11.904729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.904670 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:11.904838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.904777 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:11.904838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:11.904825 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:12.806268 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:12.806233 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" event={"ID":"27cfc3b9-7a65-4bb0-8245-ae992cde49d0","Type":"ContainerStarted","Data":"17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e"} Nov 02 03:27:12.806674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:12.806291 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" event={"ID":"27cfc3b9-7a65-4bb0-8245-ae992cde49d0","Type":"ContainerStarted","Data":"07aef3ab26ab7fff295017a10a73196bd68223ef6ca4187f030874f0947161ad"} Nov 02 03:27:12.996377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:12.995367 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podStartSLOduration=1.995328046 podStartE2EDuration="1.995328046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:12.978247702 +0000 UTC m=+2374.973948643" watchObservedRunningTime="2026-11-02 03:27:12.995328046 +0000 UTC m=+2374.991028991" Nov 02 03:27:13.435374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:13.431428 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:13.464499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:13.464466 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:13.829065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:13.828979 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:13.830687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:13.830663 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:27:15.141136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.141098 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186449 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186505 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186539 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186568 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186626 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186693 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186761 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186788 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.186977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.186874 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data\") pod \"f377d42f-1621-4215-8b24-f98dd241e830\" (UID: \"f377d42f-1621-4215-8b24-f98dd241e830\") " Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187018 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187117 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187234 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187308 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187557 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187578 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187594 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.187795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187609 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.188201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.187918 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:15.190347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.190321 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:15.191809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.191779 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:27:15.192665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.192642 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "kube-api-access-s9n6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:15.192919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.192892 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy" pod "f377d42f-1621-4215-8b24-f98dd241e830" (UID: "f377d42f-1621-4215-8b24-f98dd241e830"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.214821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.214802 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:27:15.216624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.216605 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:27:15.288513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288488 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.288646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288534 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.288646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288562 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.288646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288598 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.288646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288627 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.288872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288690 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.288872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288744 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.288872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288776 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.288872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288802 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.288872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288844 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288894 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288930 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288936 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.288988 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289038 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert\") pod \"7f23275a-45ae-4424-8aca-b9af430072bb\" (UID: \"7f23275a-45ae-4424-8aca-b9af430072bb\") " Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289053 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289069 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.289127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289102 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.289549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289164 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.289549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289201 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket\") pod \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\" (UID: \"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d\") " Nov 02 03:27:15.289549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289285 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.290496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289741 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.290496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289895 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.290496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.289984 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:15.290496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.290404 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:15.290496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.290455 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.291068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.290647 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.291068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.290708 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.291271 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291231 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:27:15.291666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291643 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9n6d\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-kube-api-access-s9n6d\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.291748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291672 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.291748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291688 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.291748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291702 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.291748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291716 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f23275a-45ae-4424-8aca-b9af430072bb-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.291748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291732 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f377d42f-1621-4215-8b24-f98dd241e830-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.291748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291746 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291760 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f377d42f-1621-4215-8b24-f98dd241e830-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291775 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f377d42f-1621-4215-8b24-f98dd241e830-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291789 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291802 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291815 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f23275a-45ae-4424-8aca-b9af430072bb-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291830 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291844 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291856 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f377d42f-1621-4215-8b24-f98dd241e830-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291874 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.292043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.291933 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "kube-api-access-t48lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:15.292524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.292202 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "kube-api-access-6p8zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:15.292734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.292710 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.293252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.293211 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:27:15.294014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.293991 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:15.294863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.294834 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy" pod "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" (UID: "6cfc71d5-9cdd-4ec8-8314-c0924e0a997d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:15.300281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.300252 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token" pod "7f23275a-45ae-4424-8aca-b9af430072bb" (UID: "7f23275a-45ae-4424-8aca-b9af430072bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:15.399451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.396318 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.399451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.396379 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p8zp\" (UniqueName: \"kubernetes.io/projected/7f23275a-45ae-4424-8aca-b9af430072bb-kube-api-access-6p8zp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.399451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.396399 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f23275a-45ae-4424-8aca-b9af430072bb-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.399451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.396414 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.399451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.396428 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t48lw\" (UniqueName: \"kubernetes.io/projected/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-kube-api-access-t48lw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.410530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.410437 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.410530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.410494 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:15.842695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.842601 4868 generic.go:356] "Generic (PLEG): container finished" podID="f377d42f-1621-4215-8b24-f98dd241e830" containerID="6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4" exitCode=0 Nov 02 03:27:15.842695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.842652 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" Nov 02 03:27:15.842935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.842685 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" event={"ID":"f377d42f-1621-4215-8b24-f98dd241e830","Type":"ContainerDied","Data":"6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4"} Nov 02 03:27:15.842935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.842746 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc" event={"ID":"f377d42f-1621-4215-8b24-f98dd241e830","Type":"ContainerDied","Data":"f2659db82f046bed1b5a07e88d60ddd346a654411904169b4c8adc7c19a2bb32"} Nov 02 03:27:15.842935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.842778 4868 scope.go:122] "RemoveContainer" containerID="6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4" Nov 02 03:27:15.844321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.844298 4868 generic.go:356] "Generic (PLEG): container finished" podID="7f23275a-45ae-4424-8aca-b9af430072bb" containerID="fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992" exitCode=0 Nov 02 03:27:15.844484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.844373 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" Nov 02 03:27:15.844484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.844417 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" event={"ID":"7f23275a-45ae-4424-8aca-b9af430072bb","Type":"ContainerDied","Data":"fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992"} Nov 02 03:27:15.844484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.844458 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb" event={"ID":"7f23275a-45ae-4424-8aca-b9af430072bb","Type":"ContainerDied","Data":"cca899210c5dd2a6c6b830c083ce4b9eb073dedeb8d39c003c7b243aa6f66314"} Nov 02 03:27:15.846330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.846307 4868 generic.go:356] "Generic (PLEG): container finished" podID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerID="8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3" exitCode=0 Nov 02 03:27:15.846487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.846383 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" event={"ID":"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d","Type":"ContainerDied","Data":"8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3"} Nov 02 03:27:15.846487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.846425 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" event={"ID":"6cfc71d5-9cdd-4ec8-8314-c0924e0a997d","Type":"ContainerDied","Data":"1b4cd8ab9fcb202094c4e17242b126c9721177efd1854146089ae10cbba490c4"} Nov 02 03:27:15.846487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.846470 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54" Nov 02 03:27:15.867567 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.867551 4868 scope.go:122] "RemoveContainer" containerID="6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4" Nov 02 03:27:15.867871 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:15.867846 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4\": container with ID starting with 6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4 not found: ID does not exist" containerID="6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4" Nov 02 03:27:15.867969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.867876 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4"} err="failed to get container status \"6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4\": rpc error: code = NotFound desc = could not find container \"6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4\": container with ID starting with 6fbfc4a33c7f04ea2c3060bb22b91c23adfb4482713de0c2d873cabe75ce62c4 not found: ID does not exist" Nov 02 03:27:15.867969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.867893 4868 scope.go:122] "RemoveContainer" containerID="fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992" Nov 02 03:27:15.876297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.876279 4868 scope.go:122] "RemoveContainer" containerID="fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992" Nov 02 03:27:15.876582 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:15.876563 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992\": container with ID starting with fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992 not found: ID does not exist" containerID="fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992" Nov 02 03:27:15.876650 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.876588 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992"} err="failed to get container status \"fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992\": rpc error: code = NotFound desc = could not find container \"fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992\": container with ID starting with fc4295ffb0b24dbf3c0c003126a13be01e63d7b2a5440c870537c661124e1992 not found: ID does not exist" Nov 02 03:27:15.876650 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.876603 4868 scope.go:122] "RemoveContainer" containerID="8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3" Nov 02 03:27:15.884708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.884690 4868 scope.go:122] "RemoveContainer" containerID="8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3" Nov 02 03:27:15.884949 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:15.884931 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3\": container with ID starting with 8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3 not found: ID does not exist" containerID="8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3" Nov 02 03:27:15.885005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:15.884954 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3"} err="failed to get container status \"8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3\": rpc error: code = NotFound desc = could not find container \"8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3\": container with ID starting with 8f9631b535141a3f8222726de5c53c45f6ed707133e9e38fc5c996e991f792e3 not found: ID does not exist" Nov 02 03:27:16.005302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.005270 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb"] Nov 02 03:27:16.010285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.010259 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q97pb"] Nov 02 03:27:16.116222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.116192 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" path="/var/lib/kubelet/pods/7f23275a-45ae-4424-8aca-b9af430072bb/volumes" Nov 02 03:27:16.119153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.119112 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54"] Nov 02 03:27:16.123286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.123259 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-wrf54"] Nov 02 03:27:16.229573 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.229545 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc"] Nov 02 03:27:16.236512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:16.236485 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dcca72-openshift-default-7c7c99b4c7-q48lc"] Nov 02 03:27:18.114873 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:18.114838 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" path="/var/lib/kubelet/pods/6cfc71d5-9cdd-4ec8-8314-c0924e0a997d/volumes" Nov 02 03:27:18.115459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:18.115442 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f377d42f-1621-4215-8b24-f98dd241e830" path="/var/lib/kubelet/pods/f377d42f-1621-4215-8b24-f98dd241e830/volumes" Nov 02 03:27:19.673508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673433 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht"] Nov 02 03:27:19.673897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673882 4868 cpu_manager.go:414] "containerMap: removing container" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" Nov 02 03:27:19.673897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673894 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" Nov 02 03:27:19.673966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673903 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" Nov 02 03:27:19.673966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673907 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" Nov 02 03:27:19.673966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673925 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" Nov 02 03:27:19.673966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673931 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" Nov 02 03:27:19.674084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.673998 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfc71d5-9cdd-4ec8-8314-c0924e0a997d" containerName="istio-proxy" Nov 02 03:27:19.674084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.674006 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f377d42f-1621-4215-8b24-f98dd241e830" containerName="istio-proxy" Nov 02 03:27:19.674084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.674014 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f23275a-45ae-4424-8aca-b9af430072bb" containerName="istio-proxy" Nov 02 03:27:19.678474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.678454 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.761733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.757323 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.761733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.757538 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7gt\" (UniqueName: \"kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt\") pod \"payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.814385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.813931 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a5285-dockercfg-m6fn4\"" Nov 02 03:27:19.814385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.814000 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a5285\"" Nov 02 03:27:19.844398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.840291 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht"] Nov 02 03:27:19.859026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.858995 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7gt\" (UniqueName: \"kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt\") pod \"payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.859184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.859067 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.859841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.859813 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.865706 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.865685 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb"] Nov 02 03:27:19.869505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.869485 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:19.890975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.890953 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb"] Nov 02 03:27:19.915989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.915960 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7gt\" (UniqueName: \"kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt\") pod \"payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:19.959880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.959811 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8n9\" (UniqueName: \"kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9\") pod \"payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:19.959880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.959861 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:19.991965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:19.991937 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:20.061172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.061090 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8n9\" (UniqueName: \"kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9\") pod \"payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:20.061287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.061182 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:20.062230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.062183 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:20.117044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.117016 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8n9\" (UniqueName: \"kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9\") pod \"payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:20.183498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.183470 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:20.364863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.364840 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht"] Nov 02 03:27:20.366242 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:20.366216 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb93c010_f500_49fb_ae98_f49c9389177a.slice/crio-167f167d530d108040e0cd6e42c9129d2c79c40fecfec4b37aa641e80d76910a WatchSource:0}: Error finding container 167f167d530d108040e0cd6e42c9129d2c79c40fecfec4b37aa641e80d76910a: Status 404 returned error can't find the container with id 167f167d530d108040e0cd6e42c9129d2c79c40fecfec4b37aa641e80d76910a Nov 02 03:27:20.556279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.556254 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb"] Nov 02 03:27:20.558166 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:20.558141 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c79e8c4_9e07_4f75_a472_367df7ec9125.slice/crio-166fb105f722ba468d193b539f1439949ab8cf86b1022a72759884b7d9644603 WatchSource:0}: Error finding container 166fb105f722ba468d193b539f1439949ab8cf86b1022a72759884b7d9644603: Status 404 returned error can't find the container with id 166fb105f722ba468d193b539f1439949ab8cf86b1022a72759884b7d9644603 Nov 02 03:27:20.792988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.792947 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch"] Nov 02 03:27:20.793480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.793229 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" containerID="cri-o://0ea29f115211327e3f12012a7f971e46e2d518e058b0acc564535b043791e5d8" gracePeriod=30 Nov 02 03:27:20.879714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.879384 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" event={"ID":"3c79e8c4-9e07-4f75-a472-367df7ec9125","Type":"ContainerStarted","Data":"823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c"} Nov 02 03:27:20.879714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.879471 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" event={"ID":"3c79e8c4-9e07-4f75-a472-367df7ec9125","Type":"ContainerStarted","Data":"166fb105f722ba468d193b539f1439949ab8cf86b1022a72759884b7d9644603"} Nov 02 03:27:20.880018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.879991 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:20.881751 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.881716 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" event={"ID":"bb93c010-f500-49fb-ae98-f49c9389177a","Type":"ContainerStarted","Data":"681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82"} Nov 02 03:27:20.881859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.881775 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" event={"ID":"bb93c010-f500-49fb-ae98-f49c9389177a","Type":"ContainerStarted","Data":"167f167d530d108040e0cd6e42c9129d2c79c40fecfec4b37aa641e80d76910a"} Nov 02 03:27:20.881990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:20.881961 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:21.025866 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:21.025700 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" podStartSLOduration=2.025681396 podStartE2EDuration="2.025681396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:21.024718642 +0000 UTC m=+2383.020419584" watchObservedRunningTime="2026-11-02 03:27:21.025681396 +0000 UTC m=+2383.021382340" Nov 02 03:27:21.139969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:21.139877 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" podStartSLOduration=2.139862343 podStartE2EDuration="2.139862343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:21.136736219 +0000 UTC m=+2383.132437162" watchObservedRunningTime="2026-11-02 03:27:21.139862343 +0000 UTC m=+2383.135563328" Nov 02 03:27:21.501999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:21.501926 4868 patch_prober.go:29] interesting pod/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:27:21.502128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:21.501991 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:27:22.589027 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:22.588996 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn"] Nov 02 03:27:22.589466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:22.589244 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" containerID="cri-o://7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6" gracePeriod=30 Nov 02 03:27:25.912307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:25.912278 4868 generic.go:356] "Generic (PLEG): container finished" podID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerID="0ea29f115211327e3f12012a7f971e46e2d518e058b0acc564535b043791e5d8" exitCode=0 Nov 02 03:27:25.912685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:25.912345 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" event={"ID":"7a23d3ea-88c3-4d61-a544-d18c549e1aaf","Type":"ContainerDied","Data":"0ea29f115211327e3f12012a7f971e46e2d518e058b0acc564535b043791e5d8"} Nov 02 03:27:25.912685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:25.912456 4868 scope.go:122] "RemoveContainer" containerID="2b00e0135188ed33b77912b9adb5a0d6b6c067d068cd550338347281c64935be" Nov 02 03:27:26.058105 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.058080 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:27:26.130509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130484 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130522 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130566 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130686 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130717 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130750 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130904 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.130987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130940 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.131041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130982 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket\") pod \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\" (UID: \"7a23d3ea-88c3-4d61-a544-d18c549e1aaf\") " Nov 02 03:27:26.131472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.130977 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:26.131614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.131161 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:26.131614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.131368 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:26.131614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.131371 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:26.131778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.131382 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:26.132809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.132782 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:27:26.132924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.132900 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:26.133066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.133043 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:26.133160 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.133060 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65" pod "7a23d3ea-88c3-4d61-a544-d18c549e1aaf" (UID: "7a23d3ea-88c3-4d61-a544-d18c549e1aaf"). InnerVolumeSpecName "kube-api-access-vkv65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:26.232574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232550 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232571 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232580 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232590 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232599 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232607 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232615 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232623 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.232717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.232631 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkv65\" (UniqueName: \"kubernetes.io/projected/7a23d3ea-88c3-4d61-a544-d18c549e1aaf-kube-api-access-vkv65\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:26.919040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.919012 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" Nov 02 03:27:26.919545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.919017 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch" event={"ID":"7a23d3ea-88c3-4d61-a544-d18c549e1aaf","Type":"ContainerDied","Data":"bdf581d04d0f72dda2e44acd174f55bb33c4e2e8c074894e3b6ed0069ce054ee"} Nov 02 03:27:26.919545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:26.919140 4868 scope.go:122] "RemoveContainer" containerID="0ea29f115211327e3f12012a7f971e46e2d518e058b0acc564535b043791e5d8" Nov 02 03:27:27.131254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.131225 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch"] Nov 02 03:27:27.135738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.135716 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c9d1fc-openshift-default-5f54f45b77-t97ch"] Nov 02 03:27:27.842776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.842745 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:27:27.925661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.925602 4868 generic.go:356] "Generic (PLEG): container finished" podID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerID="7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6" exitCode=0 Nov 02 03:27:27.925996 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.925699 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" Nov 02 03:27:27.925996 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.925736 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" event={"ID":"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7","Type":"ContainerDied","Data":"7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6"} Nov 02 03:27:27.925996 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.925782 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn" event={"ID":"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7","Type":"ContainerDied","Data":"27f63dcc780ce5a926f7dc2692bab870d069adab1f702df673922486de6f7897"} Nov 02 03:27:27.925996 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.925814 4868 scope.go:122] "RemoveContainer" containerID="7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6" Nov 02 03:27:27.934941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.934923 4868 scope.go:122] "RemoveContainer" containerID="2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a" Nov 02 03:27:27.943112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.943097 4868 scope.go:122] "RemoveContainer" containerID="7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6" Nov 02 03:27:27.943399 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:27.943382 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6\": container with ID starting with 7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6 not found: ID does not exist" containerID="7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6" Nov 02 03:27:27.943473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.943412 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6"} err="failed to get container status \"7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6\": rpc error: code = NotFound desc = could not find container \"7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6\": container with ID starting with 7d0532c182e29e522b1b5778b6089b20f69f862a1841620873ee671a97270ce6 not found: ID does not exist" Nov 02 03:27:27.943473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.943434 4868 scope.go:122] "RemoveContainer" containerID="2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a" Nov 02 03:27:27.943679 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:27.943658 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a\": container with ID starting with 2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a not found: ID does not exist" containerID="2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a" Nov 02 03:27:27.943719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.943687 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a"} err="failed to get container status \"2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a\": rpc error: code = NotFound desc = could not find container \"2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a\": container with ID starting with 2e1ef8359adf871b433224b23acbb5acca1c57418c456093a9b46c998606bf0a not found: ID does not exist" Nov 02 03:27:27.951734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951714 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.951825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951766 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.951825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951815 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.951927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951858 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.951927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951890 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.952029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951940 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.952029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.951996 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.952127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952026 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.952127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952078 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket\") pod \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\" (UID: \"2c356f5e-2156-498e-a4e3-e3d3f87d1ba7\") " Nov 02 03:27:27.952239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952096 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:27.952239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952174 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:27.952407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952345 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:27.952407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952279 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:27.952602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952486 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:27.952602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952523 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:27.952602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952539 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:27.952602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952554 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:27.952602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.952565 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:27.954082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.954053 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "kube-api-access-wvsdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:27.954171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.954093 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:27.954308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.954295 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:27.954368 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:27.954325 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo" pod "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" (UID: "2c356f5e-2156-498e-a4e3-e3d3f87d1ba7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:27:28.053119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.053091 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:28.053119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.053118 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:28.053279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.053128 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsdb\" (UniqueName: \"kubernetes.io/projected/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-kube-api-access-wvsdb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:28.053279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.053138 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:28.053279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.053147 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:28.115216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.115190 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" path="/var/lib/kubelet/pods/7a23d3ea-88c3-4d61-a544-d18c549e1aaf/volumes" Nov 02 03:27:28.341095 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.341026 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn"] Nov 02 03:27:28.390399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:28.390375 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-018cfa-openshift-default-86cbf9f749-jn2gn"] Nov 02 03:27:30.114114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:30.114079 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" path="/var/lib/kubelet/pods/2c356f5e-2156-498e-a4e3-e3d3f87d1ba7/volumes" Nov 02 03:27:31.890387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:31.890336 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:27:32.895791 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:32.895765 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:27:33.248719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:33.248648 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m"] Nov 02 03:27:33.450408 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:33.450309 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:27:38.249819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.249770 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" podUID="e65e43f3-b3eb-4279-bbb8-d6a64021b15d" containerName="payload-pre-processing" containerID="cri-o://c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f" gracePeriod=25 Nov 02 03:27:38.379515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.379487 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:27:38.389025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.389004 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:27:38.514859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.514803 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:27:38.556873 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.556845 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume\") pod \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " Nov 02 03:27:38.557084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.557066 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw\" (UniqueName: \"kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw\") pod \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\" (UID: \"e65e43f3-b3eb-4279-bbb8-d6a64021b15d\") " Nov 02 03:27:38.557274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.557246 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume" pod "e65e43f3-b3eb-4279-bbb8-d6a64021b15d" (UID: "e65e43f3-b3eb-4279-bbb8-d6a64021b15d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:38.557509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.557483 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:38.559294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.559267 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw" pod "e65e43f3-b3eb-4279-bbb8-d6a64021b15d" (UID: "e65e43f3-b3eb-4279-bbb8-d6a64021b15d"). InnerVolumeSpecName "kube-api-access-fw9vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:38.658028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:38.658003 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9vw\" (UniqueName: \"kubernetes.io/projected/e65e43f3-b3eb-4279-bbb8-d6a64021b15d-kube-api-access-fw9vw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:39.003639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.003610 4868 generic.go:356] "Generic (PLEG): container finished" podID="e65e43f3-b3eb-4279-bbb8-d6a64021b15d" containerID="c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f" exitCode=0 Nov 02 03:27:39.003906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.003664 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" Nov 02 03:27:39.003906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.003684 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" event={"ID":"e65e43f3-b3eb-4279-bbb8-d6a64021b15d","Type":"ContainerDied","Data":"c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f"} Nov 02 03:27:39.003906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.003769 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m" event={"ID":"e65e43f3-b3eb-4279-bbb8-d6a64021b15d","Type":"ContainerDied","Data":"41911a9d0ab710e48f8381e895d5123cd0dc8248bc680a487545096f529f69ae"} Nov 02 03:27:39.003906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.003795 4868 scope.go:122] "RemoveContainer" containerID="c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f" Nov 02 03:27:39.013406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.013389 4868 scope.go:122] "RemoveContainer" containerID="c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f" Nov 02 03:27:39.013667 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:39.013649 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f\": container with ID starting with c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f not found: ID does not exist" containerID="c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f" Nov 02 03:27:39.013741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.013674 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f"} err="failed to get container status \"c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f\": rpc error: code = NotFound desc = could not find container \"c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f\": container with ID starting with c5e1eda99ca3d01e878fecd5ca83665d99546ed6123f011f430adc5362db474f not found: ID does not exist" Nov 02 03:27:39.165026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.165002 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m"] Nov 02 03:27:39.166988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:39.166967 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c170e9-dfb74f975-wmv5m"] Nov 02 03:27:40.114580 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:40.114548 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65e43f3-b3eb-4279-bbb8-d6a64021b15d" path="/var/lib/kubelet/pods/e65e43f3-b3eb-4279-bbb8-d6a64021b15d/volumes" Nov 02 03:27:40.418369 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:40.418289 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9"] Nov 02 03:27:40.418537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:40.418503 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" containerID="cri-o://841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6" gracePeriod=30 Nov 02 03:27:43.983399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:43.983369 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:27:43.983788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:43.983418 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:27:45.747469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.747441 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:27:45.835826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.835788 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.835854 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.835941 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836020 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836061 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836103 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:45.836133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836112 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836138 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836165 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836176 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:45.836379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836229 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data\") pod \"12f53e4e-843e-4930-81f9-329875f19c7f\" (UID: \"12f53e4e-843e-4930-81f9-329875f19c7f\") " Nov 02 03:27:45.836597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836505 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:45.836758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836712 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:45.836897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836810 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:27:45.836972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836924 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.836972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836944 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f53e4e-843e-4930-81f9-329875f19c7f-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.836972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836957 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.836972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836971 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.837191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.836984 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.838396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.838347 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:27:45.838396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.838375 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "kube-api-access-4v5kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:45.838592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.838573 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:27:45.838637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.838626 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token" pod "12f53e4e-843e-4930-81f9-329875f19c7f" (UID: "12f53e4e-843e-4930-81f9-329875f19c7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:27:45.937948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.937911 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f53e4e-843e-4930-81f9-329875f19c7f-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.937948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.937939 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v5kb\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-kube-api-access-4v5kb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.937948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.937951 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f53e4e-843e-4930-81f9-329875f19c7f-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:45.938169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:45.937960 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f53e4e-843e-4930-81f9-329875f19c7f-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:27:46.049796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.049695 4868 generic.go:356] "Generic (PLEG): container finished" podID="12f53e4e-843e-4930-81f9-329875f19c7f" containerID="841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6" exitCode=0 Nov 02 03:27:46.049796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.049764 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" Nov 02 03:27:46.050021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.049784 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" event={"ID":"12f53e4e-843e-4930-81f9-329875f19c7f","Type":"ContainerDied","Data":"841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6"} Nov 02 03:27:46.050021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.049849 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9" event={"ID":"12f53e4e-843e-4930-81f9-329875f19c7f","Type":"ContainerDied","Data":"3e65b865c086d011a30b55ffd316bb9076fe2fa13120251939137b0d235a16d3"} Nov 02 03:27:46.050021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.049878 4868 scope.go:122] "RemoveContainer" containerID="841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6" Nov 02 03:27:46.059964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.059940 4868 scope.go:122] "RemoveContainer" containerID="841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6" Nov 02 03:27:46.060251 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:46.060227 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6\": container with ID starting with 841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6 not found: ID does not exist" containerID="841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6" Nov 02 03:27:46.060383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.060259 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6"} err="failed to get container status \"841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6\": rpc error: code = NotFound desc = could not find container \"841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6\": container with ID starting with 841b5640983798998032783ff4eafbc0de2dd1f6660924ae7fba5fe54d3200e6 not found: ID does not exist" Nov 02 03:27:46.096055 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096029 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh"] Nov 02 03:27:46.096482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096469 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" Nov 02 03:27:46.096529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096484 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" Nov 02 03:27:46.096529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096506 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" Nov 02 03:27:46.096529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096511 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" Nov 02 03:27:46.096529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096524 4868 cpu_manager.go:414] "containerMap: removing container" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" Nov 02 03:27:46.096529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096529 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096542 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e65e43f3-b3eb-4279-bbb8-d6a64021b15d" containerName="payload-pre-processing" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096547 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65e43f3-b3eb-4279-bbb8-d6a64021b15d" containerName="payload-pre-processing" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096610 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096617 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" containerName="istio-proxy" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096623 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096630 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65e43f3-b3eb-4279-bbb8-d6a64021b15d" containerName="payload-pre-processing" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096637 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" Nov 02 03:27:46.096685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.096644 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" Nov 02 03:27:46.101335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.101316 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.139938 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.139997 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140056 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140087 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7s68\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140119 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140144 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140185 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140211 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.140523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.140319 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.153881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.153856 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4c3ea5-openshift-default-dockercfg-hqcdq\"" Nov 02 03:27:46.182867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.182833 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh"] Nov 02 03:27:46.217837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.217809 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9"] Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246112 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246200 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246241 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246298 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246329 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7s68\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246377 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246402 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246461 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.247044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.246487 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.286568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.286538 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.286568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.286537 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.286811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.286749 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.286914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.286897 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.287626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.287578 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.288078 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.288055 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-xlfg9"] Nov 02 03:27:46.289791 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.289765 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.289899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.289824 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.343192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.343124 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7s68\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.343464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.343444 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token\") pod \"e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.423278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.423240 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:46.845738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.845705 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh"] Nov 02 03:27:46.847914 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:46.847884 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8da2353_de4d_449c_9000_de0e4632bf69.slice/crio-c60b25db557f7d5b5e3748d94cbb6a71955177c1a9da2498a3d6ee4dd54cd5c6 WatchSource:0}: Error finding container c60b25db557f7d5b5e3748d94cbb6a71955177c1a9da2498a3d6ee4dd54cd5c6: Status 404 returned error can't find the container with id c60b25db557f7d5b5e3748d94cbb6a71955177c1a9da2498a3d6ee4dd54cd5c6 Nov 02 03:27:46.850409 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.850325 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:46.850517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.850418 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:46.850517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:46.850451 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:47.058464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:47.058375 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" event={"ID":"d8da2353-de4d-449c-9000-de0e4632bf69","Type":"ContainerStarted","Data":"7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0"} Nov 02 03:27:47.058464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:47.058441 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" event={"ID":"d8da2353-de4d-449c-9000-de0e4632bf69","Type":"ContainerStarted","Data":"c60b25db557f7d5b5e3748d94cbb6a71955177c1a9da2498a3d6ee4dd54cd5c6"} Nov 02 03:27:47.176159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:47.176095 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podStartSLOduration=1.176076532 podStartE2EDuration="1.176076532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:47.173861731 +0000 UTC m=+2409.169562676" watchObservedRunningTime="2026-11-02 03:27:47.176076532 +0000 UTC m=+2409.171777477" Nov 02 03:27:47.424345 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:47.424308 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:47.429694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:47.429625 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:48.070273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:48.070236 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:48.071156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:48.071137 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:27:48.113471 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:48.113442 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f53e4e-843e-4930-81f9-329875f19c7f" path="/var/lib/kubelet/pods/12f53e4e-843e-4930-81f9-329875f19c7f/volumes" Nov 02 03:27:51.640852 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.640818 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth"] Nov 02 03:27:51.641472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.641451 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" Nov 02 03:27:51.641649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.641629 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c356f5e-2156-498e-a4e3-e3d3f87d1ba7" containerName="istio-proxy" Nov 02 03:27:51.641776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.641678 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" Nov 02 03:27:51.641776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.641686 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a23d3ea-88c3-4d61-a544-d18c549e1aaf" containerName="istio-proxy" Nov 02 03:27:51.647123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.647104 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.700564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.700539 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth"] Nov 02 03:27:51.703432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.703407 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6a5285-serving-cert\"" Nov 02 03:27:51.811262 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.811231 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.811431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.811287 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.811431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.811401 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.811520 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.811438 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46ln\" (UniqueName: \"kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.914604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.914504 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.914604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.914601 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.914830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.914651 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.914830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.914692 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46ln\" (UniqueName: \"kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.915225 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:51.915190 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-serving-cert: secret "maas-api-e2e-api-a-6a5285-serving-cert" not found Nov 02 03:27:51.915398 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:27:51.915266 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls podName:9f199224-aa72-4c40-8d80-b5b587b8c8b3 nodeName:}" failed. No retries permitted until 2026-11-02 03:27:52.415246135 +0000 UTC m=+2414.410947069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls") pod "maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" (UID: "9f199224-aa72-4c40-8d80-b5b587b8c8b3") : secret "maas-api-e2e-api-a-6a5285-serving-cert" not found Nov 02 03:27:51.916030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.916003 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.920026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.920002 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:51.972936 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:51.972913 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46ln\" (UniqueName: \"kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:52.421082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:52.421048 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:52.423632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:52.423605 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\") pod \"maas-api-e2e-api-a-6a5285-d8954f6c-9ngth\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:52.565569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:52.565537 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:52.938130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:52.938107 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth"] Nov 02 03:27:52.940130 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:52.940105 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f199224_aa72_4c40_8d80_b5b587b8c8b3.slice/crio-4dbe162fbf0f87b552d92d6eb7ab195cbb354cd049f16ff408bc0e9e75ff4dc1 WatchSource:0}: Error finding container 4dbe162fbf0f87b552d92d6eb7ab195cbb354cd049f16ff408bc0e9e75ff4dc1: Status 404 returned error can't find the container with id 4dbe162fbf0f87b552d92d6eb7ab195cbb354cd049f16ff408bc0e9e75ff4dc1 Nov 02 03:27:52.941515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:52.941495 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:27:53.095868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:53.095840 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" event={"ID":"9f199224-aa72-4c40-8d80-b5b587b8c8b3","Type":"ContainerStarted","Data":"4dbe162fbf0f87b552d92d6eb7ab195cbb354cd049f16ff408bc0e9e75ff4dc1"} Nov 02 03:27:54.102893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:54.102853 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" event={"ID":"9f199224-aa72-4c40-8d80-b5b587b8c8b3","Type":"ContainerStarted","Data":"0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904"} Nov 02 03:27:54.103385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:54.102914 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:27:54.215590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:54.215537 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" podStartSLOduration=2.352244748 podStartE2EDuration="3.215517887s" totalImagesPullingTime="863.273139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:54.214874512 +0000 UTC m=+2416.210575460" watchObservedRunningTime="2026-11-02 03:27:54.215517887 +0000 UTC m=+2416.211218831" Nov 02 03:27:55.298224 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.298188 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls"] Nov 02 03:27:55.302700 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.302674 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.357747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.357713 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls"] Nov 02 03:27:55.364850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.364809 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4c3ea5-dockercfg-twpl4\"" Nov 02 03:27:55.365198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.365179 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4c3ea5\"" Nov 02 03:27:55.453982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.453761 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6"] Nov 02 03:27:55.464496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.464467 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.465654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.465556 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.465916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.465890 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9qd\" (UniqueName: \"kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd\") pod \"payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.516842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.516615 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6"] Nov 02 03:27:55.567201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.567113 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.567375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.567231 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.567375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.567265 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcj69\" (UniqueName: \"kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69\") pod \"payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.567375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.567321 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9qd\" (UniqueName: \"kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd\") pod \"payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.567973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.567948 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.623476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.623439 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9qd\" (UniqueName: \"kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd\") pod \"payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:55.669319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.669281 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.669484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.669421 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcj69\" (UniqueName: \"kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69\") pod \"payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.670786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.670751 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.725329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.725299 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcj69\" (UniqueName: \"kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69\") pod \"payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.795114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.795083 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:55.916240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:55.916202 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:56.165849 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:56.165820 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6"] Nov 02 03:27:56.167677 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:56.167651 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78799308_9296_4456_b7fc_4f00a6353a18.slice/crio-f1d037d833a0fd3956b0c52d65549b964063652b44e70e52c1295481d23bac1e WatchSource:0}: Error finding container f1d037d833a0fd3956b0c52d65549b964063652b44e70e52c1295481d23bac1e: Status 404 returned error can't find the container with id f1d037d833a0fd3956b0c52d65549b964063652b44e70e52c1295481d23bac1e Nov 02 03:27:56.289289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:56.289265 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls"] Nov 02 03:27:56.290805 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:56.290773 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode367ea5b_449e_4185_9f60_436900d4981f.slice/crio-a6efc49e73a418f5b94335e5a54e85f931ad8789f72c70869d82e882664ab793 WatchSource:0}: Error finding container a6efc49e73a418f5b94335e5a54e85f931ad8789f72c70869d82e882664ab793: Status 404 returned error can't find the container with id a6efc49e73a418f5b94335e5a54e85f931ad8789f72c70869d82e882664ab793 Nov 02 03:27:57.116515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.116481 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8"] Nov 02 03:27:57.120952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.120926 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.124482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.124449 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" event={"ID":"78799308-9296-4456-b7fc-4f00a6353a18","Type":"ContainerStarted","Data":"229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab"} Nov 02 03:27:57.124614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.124501 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" event={"ID":"78799308-9296-4456-b7fc-4f00a6353a18","Type":"ContainerStarted","Data":"f1d037d833a0fd3956b0c52d65549b964063652b44e70e52c1295481d23bac1e"} Nov 02 03:27:57.124614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.124532 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:27:57.126328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.126303 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" event={"ID":"e367ea5b-449e-4185-9f60-436900d4981f","Type":"ContainerStarted","Data":"a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc"} Nov 02 03:27:57.126435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.126342 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" event={"ID":"e367ea5b-449e-4185-9f60-436900d4981f","Type":"ContainerStarted","Data":"a6efc49e73a418f5b94335e5a54e85f931ad8789f72c70869d82e882664ab793"} Nov 02 03:27:57.126483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.126439 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:27:57.166253 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.166229 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7"] Nov 02 03:27:57.169815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.169796 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk"] Nov 02 03:27:57.169920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.169903 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.173412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.173393 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.213553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.213516 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tk8sg\"" Nov 02 03:27:57.219468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.219435 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8"] Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293770 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77smc\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293818 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293845 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293912 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7czw\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293943 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293975 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.293996 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.294017 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.294057 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.294081 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.295383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.294133 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.294152 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.295753 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.295795 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.295994 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296030 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d24q\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296058 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296090 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296112 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296163 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296196 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296415 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7"] Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296431 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296463 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.296491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296492 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.296843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296626 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.296933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296911 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.296996 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.296948 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.350375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.350331 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk"] Nov 02 03:27:57.406193 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.405988 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.407864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.407837 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.408573 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.408549 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.409418 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.409134 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.409768 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.409706 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.411026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411004 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d24q\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.411175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411159 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.411281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411268 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.411394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411379 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.411485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411474 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.411585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411572 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.411741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.411729 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.412227 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.412203 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.412403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.412384 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.412530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.412512 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.412666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.412651 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.412780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.412763 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.412937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.412921 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77smc\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.413057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413043 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.413162 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413148 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.413442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413301 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7czw\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.413602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413583 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.413761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413742 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.413877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413862 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.413983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.413967 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.414102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.414086 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.414205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.414191 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.414793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.414770 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.414926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.407316 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.415300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.409054 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.416110 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.410684 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.416528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.416507 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.416899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.416879 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.417268 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.417248 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.418243 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.418219 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.430549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.430105 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.435586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.434172 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.435586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.434794 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.435586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.435063 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.435586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.435108 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.435586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.435326 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.435586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.435386 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.440285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.440242 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.458996 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.451036 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.462942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.462702 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.466448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.463580 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.466448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.463757 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.467156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.467109 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.469445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.469394 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.479726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.479705 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.480283 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.480264 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d24q\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dmsh8\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.490000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.489970 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77smc\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7sb7\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.506506 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.506488 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.515122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.515104 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7czw\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nm8gk\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:57.583000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.582951 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" podStartSLOduration=2.582936806 podStartE2EDuration="2.582936806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:57.582037268 +0000 UTC m=+2419.577738212" watchObservedRunningTime="2026-11-02 03:27:57.582936806 +0000 UTC m=+2419.578637749" Nov 02 03:27:57.695426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.695018 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" podStartSLOduration=2.695000868 podStartE2EDuration="2.695000868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:57.694122214 +0000 UTC m=+2419.689823181" watchObservedRunningTime="2026-11-02 03:27:57.695000868 +0000 UTC m=+2419.690701813" Nov 02 03:27:57.736549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.736517 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:57.783566 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.783534 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:57.790156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:57.789924 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:58.161405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.161378 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8"] Nov 02 03:27:58.162479 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:58.162452 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea2374b_e26e_41e5_b53f_f57fe695ecad.slice/crio-dc15fc02b372cddba040837b4dfd6ca7a42eaf222db93cc973aa711743942bdd WatchSource:0}: Error finding container dc15fc02b372cddba040837b4dfd6ca7a42eaf222db93cc973aa711743942bdd: Status 404 returned error can't find the container with id dc15fc02b372cddba040837b4dfd6ca7a42eaf222db93cc973aa711743942bdd Nov 02 03:27:58.165125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.165094 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.165200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.165170 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.165240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.165212 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.286888 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.284562 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk"] Nov 02 03:27:58.288775 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.287650 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7"] Nov 02 03:27:58.289165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.289130 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.289265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.289195 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.289265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.289237 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.289587 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:27:58.289563 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b1a546_10f6_4e88_bbae_492093fdc38d.slice/crio-8e9a5a6fab0236cd7f472c0604062671cc4bff12c0d53eebf3635e82daf3a35c WatchSource:0}: Error finding container 8e9a5a6fab0236cd7f472c0604062671cc4bff12c0d53eebf3635e82daf3a35c: Status 404 returned error can't find the container with id 8e9a5a6fab0236cd7f472c0604062671cc4bff12c0d53eebf3635e82daf3a35c Nov 02 03:27:58.292039 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.292013 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.292105 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.292075 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:58.292158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:58.292104 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:27:59.198373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.197318 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" event={"ID":"2ea2374b-e26e-41e5-b53f-f57fe695ecad","Type":"ContainerStarted","Data":"680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69"} Nov 02 03:27:59.198373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.197418 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" event={"ID":"2ea2374b-e26e-41e5-b53f-f57fe695ecad","Type":"ContainerStarted","Data":"dc15fc02b372cddba040837b4dfd6ca7a42eaf222db93cc973aa711743942bdd"} Nov 02 03:27:59.205711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.205677 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" event={"ID":"14b1a546-10f6-4e88-bbae-492093fdc38d","Type":"ContainerStarted","Data":"9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b"} Nov 02 03:27:59.205978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.205957 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" event={"ID":"14b1a546-10f6-4e88-bbae-492093fdc38d","Type":"ContainerStarted","Data":"8e9a5a6fab0236cd7f472c0604062671cc4bff12c0d53eebf3635e82daf3a35c"} Nov 02 03:27:59.209612 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.209584 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" event={"ID":"6e8dfdf1-69f9-418d-9455-75533b89a6ea","Type":"ContainerStarted","Data":"95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7"} Nov 02 03:27:59.209894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.209874 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" event={"ID":"6e8dfdf1-69f9-418d-9455-75533b89a6ea","Type":"ContainerStarted","Data":"ab8ecf7b5482614cd38ef37fd283637634dcf14f7efbad3c70fc15351b4231cb"} Nov 02 03:27:59.320278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.320229 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podStartSLOduration=2.320216004 podStartE2EDuration="2.320216004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:59.317727439 +0000 UTC m=+2421.313428368" watchObservedRunningTime="2026-11-02 03:27:59.320216004 +0000 UTC m=+2421.315916948" Nov 02 03:27:59.572436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.572149 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podStartSLOduration=2.572080021 podStartE2EDuration="2.572080021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:59.447628641 +0000 UTC m=+2421.443329584" watchObservedRunningTime="2026-11-02 03:27:59.572080021 +0000 UTC m=+2421.567780965" Nov 02 03:27:59.746377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.739690 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:27:59.757384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.756496 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:27:59.757384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.756568 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:27:59.803378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.788172 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:27:59.803378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.788327 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:27:59.803378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.788396 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:27:59.803378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.793405 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:27:59.803378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.797449 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:27:59.803378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:27:59.797510 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:00.123331 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.123300 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:28:00.263378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.255650 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podStartSLOduration=3.255629011 podStartE2EDuration="3.255629011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:27:59.575672956 +0000 UTC m=+2421.571373900" watchObservedRunningTime="2026-11-02 03:28:00.255629011 +0000 UTC m=+2422.251329955" Nov 02 03:28:00.744631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.744596 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:28:00.744927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.744898 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:28:00.788798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.788758 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:28:00.789082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.789049 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:28:00.791710 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.791685 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:28:00.791881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:00.791860 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:01.779387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:01.769544 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:28:01.779387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:01.769644 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:28:01.785958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:01.785809 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:28:01.785958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:01.785893 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:28:01.800576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:01.800456 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:28:01.800576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:01.800518 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:02.739377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:02.738688 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:28:02.739377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:02.738749 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:28:02.788458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:02.787791 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:28:02.788458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:02.787854 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:28:02.792373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:02.791713 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:28:02.792373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:02.791768 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:03.741375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:03.737715 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:28:03.792476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:03.792035 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:28:03.793033 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:03.793004 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:28:03.793139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:03.792338 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:28:03.793226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:03.793212 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:03.793338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:03.792430 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:28:03.982384 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:28:03.978610 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:28:04.737766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:04.737725 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:28:04.738052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:04.738019 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:28:04.784631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:04.784585 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:28:04.784804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:04.784656 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:28:04.791036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:04.791007 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:28:04.791103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:04.791061 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:05.737809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:05.737770 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 03:28:05.738206 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:05.737849 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 03:28:05.785087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:05.785055 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7sb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 03:28:05.785227 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:05.785109 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 03:28:05.790457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:05.790430 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 03:28:05.790584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:05.790502 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 03:28:07.312377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.312159 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:28:07.313502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.313416 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:28:07.314211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.314186 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:28:07.368438 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.368400 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:28:07.368849 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.368813 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:28:07.369687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.369668 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:28:07.635339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.635279 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:28:07.636645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.636620 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:28:07.642509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:07.642486 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:28:08.137431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:08.137402 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 03:28:08.137733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:08.137704 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 03:28:13.112373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:13.111495 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:28:13.238429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:13.234926 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:28:20.498142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:20.498113 4868 generic.go:356] "Generic (PLEG): container finished" podID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerID="17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e" exitCode=0 Nov 02 03:28:20.498497 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:20.498163 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" event={"ID":"27cfc3b9-7a65-4bb0-8245-ae992cde49d0","Type":"ContainerDied","Data":"17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e"} Nov 02 03:28:20.498554 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:20.498538 4868 scope.go:122] "RemoveContainer" containerID="17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e" Nov 02 03:28:20.500023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:20.499994 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:20.500120 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:20.500062 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:20.500120 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:20.500093 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:21.507305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:21.507267 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" event={"ID":"27cfc3b9-7a65-4bb0-8245-ae992cde49d0","Type":"ContainerStarted","Data":"89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13"} Nov 02 03:28:22.420566 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:22.420531 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:28:22.420743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:22.420710 4868 patch_prober.go:29] interesting pod/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 03:28:22.420789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:22.420762 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 03:28:23.420846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:23.420805 4868 patch_prober.go:29] interesting pod/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 03:28:23.421239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:23.420863 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 03:28:24.421315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:24.421280 4868 patch_prober.go:29] interesting pod/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 03:28:24.421750 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:24.421335 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 03:28:25.428380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:25.427657 4868 patch_prober.go:29] interesting pod/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 03:28:25.428380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:25.427733 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 03:28:26.420793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:26.420753 4868 patch_prober.go:29] interesting pod/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 03:28:26.421004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:26.420822 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 03:28:27.425682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:27.425650 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:28:27.426177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:27.426072 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:28:27.426801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:27.426768 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:28:33.449528 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:28:33.449495 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:28:37.625391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:37.623175 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq"] Nov 02 03:28:37.625391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:37.623535 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" podUID="3cbdf351-b349-442a-8226-ebbb05d0bacd" containerName="maas-api" containerID="cri-o://2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d" gracePeriod=30 Nov 02 03:28:38.090087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.090049 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:28:38.232267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.232177 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\") pod \"3cbdf351-b349-442a-8226-ebbb05d0bacd\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " Nov 02 03:28:38.232448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.232386 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca\") pod \"3cbdf351-b349-442a-8226-ebbb05d0bacd\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " Nov 02 03:28:38.232448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.232424 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs\") pod \"3cbdf351-b349-442a-8226-ebbb05d0bacd\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " Nov 02 03:28:38.232585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.232568 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876\" (UniqueName: \"kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876\") pod \"3cbdf351-b349-442a-8226-ebbb05d0bacd\" (UID: \"3cbdf351-b349-442a-8226-ebbb05d0bacd\") " Nov 02 03:28:38.232785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.232756 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca" pod "3cbdf351-b349-442a-8226-ebbb05d0bacd" (UID: "3cbdf351-b349-442a-8226-ebbb05d0bacd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:28:38.233428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.233400 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbdf351-b349-442a-8226-ebbb05d0bacd-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:28:38.234773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.234749 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls" pod "3cbdf351-b349-442a-8226-ebbb05d0bacd" (UID: "3cbdf351-b349-442a-8226-ebbb05d0bacd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:28:38.235248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.235223 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs" pod "3cbdf351-b349-442a-8226-ebbb05d0bacd" (UID: "3cbdf351-b349-442a-8226-ebbb05d0bacd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:28:38.235369 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.235268 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876" pod "3cbdf351-b349-442a-8226-ebbb05d0bacd" (UID: "3cbdf351-b349-442a-8226-ebbb05d0bacd"). InnerVolumeSpecName "kube-api-access-bf876". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:28:38.334898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.334867 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:28:38.334898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.334893 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf876\" (UniqueName: \"kubernetes.io/projected/3cbdf351-b349-442a-8226-ebbb05d0bacd-kube-api-access-bf876\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:28:38.334898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.334904 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cbdf351-b349-442a-8226-ebbb05d0bacd-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:28:38.667663 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.667632 4868 generic.go:356] "Generic (PLEG): container finished" podID="3cbdf351-b349-442a-8226-ebbb05d0bacd" containerID="2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d" exitCode=0 Nov 02 03:28:38.668060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.667678 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" Nov 02 03:28:38.668060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.667686 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" event={"ID":"3cbdf351-b349-442a-8226-ebbb05d0bacd","Type":"ContainerDied","Data":"2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d"} Nov 02 03:28:38.668060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.667726 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq" event={"ID":"3cbdf351-b349-442a-8226-ebbb05d0bacd","Type":"ContainerDied","Data":"ba8884a2815e3e72685599b574aafb6e3dcb82dce944ff2d26ffa8bfa810c545"} Nov 02 03:28:38.668060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.667747 4868 scope.go:122] "RemoveContainer" containerID="2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d" Nov 02 03:28:38.683555 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.683534 4868 scope.go:122] "RemoveContainer" containerID="2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d" Nov 02 03:28:38.683929 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:28:38.683896 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d\": container with ID starting with 2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d not found: ID does not exist" containerID="2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d" Nov 02 03:28:38.684013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.683949 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d"} err="failed to get container status \"2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d\": rpc error: code = NotFound desc = could not find container \"2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d\": container with ID starting with 2e9c28075bbc29849f3c9900a35bcbcedf7ed4bf395f902122fb12bca0ce4f6d not found: ID does not exist" Nov 02 03:28:38.829789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.829756 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq"] Nov 02 03:28:38.879098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:38.879021 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c170e9-6dd99588b8-psdmq"] Nov 02 03:28:40.116156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:40.116123 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cbdf351-b349-442a-8226-ebbb05d0bacd" path="/var/lib/kubelet/pods/3cbdf351-b349-442a-8226-ebbb05d0bacd/volumes" Nov 02 03:28:43.341196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.341162 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9"] Nov 02 03:28:43.341657 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.341643 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3cbdf351-b349-442a-8226-ebbb05d0bacd" containerName="maas-api" Nov 02 03:28:43.341701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.341657 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cbdf351-b349-442a-8226-ebbb05d0bacd" containerName="maas-api" Nov 02 03:28:43.341776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.341763 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cbdf351-b349-442a-8226-ebbb05d0bacd" containerName="maas-api" Nov 02 03:28:43.345222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.345201 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.405376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.403727 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f1a692-openshift-default-dockercfg-vbjtb\"" Nov 02 03:28:43.465400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.465345 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9"] Nov 02 03:28:43.494061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.494016 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.494456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.494344 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.494691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.494667 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.503372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.494838 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.503631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.503606 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.503753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.503738 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.503850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.503834 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrk4n\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.504039 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.504025 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.504180 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.504166 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605224 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605184 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605246 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605329 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605388 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605520 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605439 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605520 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605493 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605598 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605534 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605598 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605565 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.605667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605602 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrk4n\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.606220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.605999 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.606220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.606064 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.606220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.606192 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.606436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.606382 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.606504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.606487 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.608392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.608369 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.608619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.608598 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.662074 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.662045 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.662234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.662196 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrk4n\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n\") pod \"e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:43.672909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:43.672888 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:44.160081 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.160056 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9"] Nov 02 03:28:44.161775 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:28:44.161742 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ecbe05_9f48_4b2a_8654_0823907a23aa.slice/crio-a01dc3847ee76f35fc88d7e6a068128659a40db7a1dcaa6158b365c43988e307 WatchSource:0}: Error finding container a01dc3847ee76f35fc88d7e6a068128659a40db7a1dcaa6158b365c43988e307: Status 404 returned error can't find the container with id a01dc3847ee76f35fc88d7e6a068128659a40db7a1dcaa6158b365c43988e307 Nov 02 03:28:44.164741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.164709 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:44.164831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.164771 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:44.164831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.164801 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:44.727065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.727024 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" event={"ID":"f7ecbe05-9f48-4b2a-8654-0823907a23aa","Type":"ContainerStarted","Data":"2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478"} Nov 02 03:28:44.727539 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.727082 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" event={"ID":"f7ecbe05-9f48-4b2a-8654-0823907a23aa","Type":"ContainerStarted","Data":"a01dc3847ee76f35fc88d7e6a068128659a40db7a1dcaa6158b365c43988e307"} Nov 02 03:28:44.863448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:44.863395 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podStartSLOduration=1.8633760179999999 podStartE2EDuration="1.863376018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:28:44.861485639 +0000 UTC m=+2466.857186599" watchObservedRunningTime="2026-11-02 03:28:44.863376018 +0000 UTC m=+2466.859076965" Nov 02 03:28:45.673243 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:45.673209 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:45.678195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:45.678172 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:45.732878 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:45.732852 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:45.733660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:45.733641 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:28:48.029492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.029457 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs"] Nov 02 03:28:48.033171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.033154 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.117458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.117425 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs"] Nov 02 03:28:48.141847 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.141823 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-abc348-openshift-default-dockercfg-cvh67\"" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.179621 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.179676 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.179772 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.179867 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.179893 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.179982 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.180013 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.180056 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwx7d\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.183564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.180108 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283024 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283081 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283184 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283296 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283324 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283467 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283492 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283533 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwx7d\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.284426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.283568 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.285083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.284744 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.291159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.285540 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.291159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.289045 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.291159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.289791 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.291159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.293758 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.291159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.299750 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.291159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.300939 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.350073 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.350041 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.353304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.353270 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwx7d\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d\") pod \"e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:48.647604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:48.647570 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:49.082619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.082592 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs"] Nov 02 03:28:49.090787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.090753 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:49.090882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.090820 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:49.090882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.090869 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:49.858393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.851283 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" event={"ID":"a9a68900-05c8-4e9c-b89d-ef751561e4f7","Type":"ContainerStarted","Data":"f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8"} Nov 02 03:28:49.858393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.851373 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" event={"ID":"a9a68900-05c8-4e9c-b89d-ef751561e4f7","Type":"ContainerStarted","Data":"603e610dae086d53e24832928686d68a21fa146cca9c915de99586450a711e24"} Nov 02 03:28:49.967835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:49.967773 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podStartSLOduration=2.967757507 podStartE2EDuration="2.967757507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:28:49.9674829 +0000 UTC m=+2471.963183843" watchObservedRunningTime="2026-11-02 03:28:49.967757507 +0000 UTC m=+2471.963458450" Nov 02 03:28:50.653375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:50.652566 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:50.666370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:50.663612 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:50.860909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:50.860874 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:50.861966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:50.861940 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:28:58.907597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:58.907565 4868 generic.go:356] "Generic (PLEG): container finished" podID="d8da2353-de4d-449c-9000-de0e4632bf69" containerID="7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0" exitCode=0 Nov 02 03:28:58.908029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:58.907642 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" event={"ID":"d8da2353-de4d-449c-9000-de0e4632bf69","Type":"ContainerDied","Data":"7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0"} Nov 02 03:28:58.908127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:58.908108 4868 scope.go:122] "RemoveContainer" containerID="7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0" Nov 02 03:28:58.909748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:58.909713 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:58.909825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:58.909792 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:58.909861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:58.909825 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:28:59.916173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:28:59.916138 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" event={"ID":"d8da2353-de4d-449c-9000-de0e4632bf69","Type":"ContainerStarted","Data":"35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca"} Nov 02 03:29:00.423529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:00.423498 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:29:00.424950 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:00.424922 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 03:29:00.425146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:00.424980 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 03:29:01.424475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:01.424438 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 03:29:01.424877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:01.424496 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 03:29:02.423904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:02.423868 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 03:29:02.424084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:02.423921 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 03:29:03.070892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:03.070857 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:29:03.424385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:03.424326 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 03:29:03.424578 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:03.424399 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 03:29:03.468401 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:03.468331 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:29:04.423818 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:04.423778 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 03:29:04.424208 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:04.423836 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 03:29:05.428027 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:05.427999 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:29:05.429016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:05.428996 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 03:29:10.131410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.131374 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877"] Nov 02 03:29:10.136133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.136103 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.190518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.190491 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-045653-serving-cert\"" Nov 02 03:29:10.242197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.242161 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877"] Nov 02 03:29:10.269455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.269417 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7zp\" (UniqueName: \"kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.269609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.269521 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.269609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.269539 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.269609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.269587 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.404877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.388176 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.405191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.396196 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.405191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.405093 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.405335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.405199 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.405335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.405318 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7zp\" (UniqueName: \"kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.405836 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:10.405812 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-serving-cert: secret "maas-api-e2e-ipp-a-045653-serving-cert" not found Nov 02 03:29:10.406044 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:10.406027 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls podName:7dc4386b-1062-4e33-aff2-a14067e6831e nodeName:}" failed. No retries permitted until 2026-11-02 03:29:10.906001807 +0000 UTC m=+2492.901702746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls") pod "maas-api-e2e-ipp-a-045653-88cd885cb-ww877" (UID: "7dc4386b-1062-4e33-aff2-a14067e6831e") : secret "maas-api-e2e-ipp-a-045653-serving-cert" not found Nov 02 03:29:10.418022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.417924 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.462504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.462471 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7zp\" (UniqueName: \"kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.911113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.911071 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:10.913562 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:10.913526 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-045653-88cd885cb-ww877\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:11.058391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:11.058340 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:11.439630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:11.439601 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877"] Nov 02 03:29:11.440172 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:29:11.440148 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc4386b_1062_4e33_aff2_a14067e6831e.slice/crio-302a6a18c872747240c53ec609010dc461dd5280c799fb62a10817226b4d3e93 WatchSource:0}: Error finding container 302a6a18c872747240c53ec609010dc461dd5280c799fb62a10817226b4d3e93: Status 404 returned error can't find the container with id 302a6a18c872747240c53ec609010dc461dd5280c799fb62a10817226b4d3e93 Nov 02 03:29:12.032478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:12.028513 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" event={"ID":"7dc4386b-1062-4e33-aff2-a14067e6831e","Type":"ContainerStarted","Data":"302a6a18c872747240c53ec609010dc461dd5280c799fb62a10817226b4d3e93"} Nov 02 03:29:13.036476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:13.036442 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" event={"ID":"7dc4386b-1062-4e33-aff2-a14067e6831e","Type":"ContainerStarted","Data":"eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de"} Nov 02 03:29:13.036886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:13.036504 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:13.150618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:13.150568 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" podStartSLOduration=2.306428525 podStartE2EDuration="3.150553367s" totalImagesPullingTime="844.124842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:13.149317084 +0000 UTC m=+2495.145018027" watchObservedRunningTime="2026-11-02 03:29:13.150553367 +0000 UTC m=+2495.146254310" Nov 02 03:29:19.047258 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:19.047229 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:29:25.258889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:25.258858 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc"] Nov 02 03:29:25.259396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:25.259059 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" containerID="cri-o://e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73" gracePeriod=30 Nov 02 03:29:25.412751 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:25.412718 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:29:25.412911 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:25.412782 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:29:30.572289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.572262 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:29:30.664577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664549 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.664744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664592 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.664744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664644 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.664744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664672 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.664931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664860 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.664931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664903 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:29:30.665059 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.664993 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.665059 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665037 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.665168 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665078 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:29:30.665168 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665098 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.665279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665190 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket\") pod \"03e7f592-6cca-4fff-8711-ea3f7f39a578\" (UID: \"03e7f592-6cca-4fff-8711-ea3f7f39a578\") " Nov 02 03:29:30.665279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665237 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:29:30.665411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665282 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:29:30.665579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665548 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:29:30.665759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665734 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.665837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665760 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e7f592-6cca-4fff-8711-ea3f7f39a578-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.665837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665776 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.665837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665791 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.665837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.665805 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.666975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.666948 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:29:30.667079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.667018 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:29:30.667079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.667038 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "kube-api-access-tljwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:29:30.667196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.667178 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo" pod "03e7f592-6cca-4fff-8711-ea3f7f39a578" (UID: "03e7f592-6cca-4fff-8711-ea3f7f39a578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:29:30.767121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.767088 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.767121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.767116 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.767292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.767130 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljwx\" (UniqueName: \"kubernetes.io/projected/03e7f592-6cca-4fff-8711-ea3f7f39a578-kube-api-access-tljwx\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:30.767292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:30.767144 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e7f592-6cca-4fff-8711-ea3f7f39a578-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:29:31.162390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.162341 4868 generic.go:356] "Generic (PLEG): container finished" podID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerID="e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73" exitCode=0 Nov 02 03:29:31.162590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.162439 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" Nov 02 03:29:31.162590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.162456 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" event={"ID":"03e7f592-6cca-4fff-8711-ea3f7f39a578","Type":"ContainerDied","Data":"e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73"} Nov 02 03:29:31.162590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.162523 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc" event={"ID":"03e7f592-6cca-4fff-8711-ea3f7f39a578","Type":"ContainerDied","Data":"e64fc1f6a3f4b315047d392c90de92e4fff65bee12f07b3fa976bb96ad3e5875"} Nov 02 03:29:31.162590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.162554 4868 scope.go:122] "RemoveContainer" containerID="e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73" Nov 02 03:29:31.173032 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.173010 4868 scope.go:122] "RemoveContainer" containerID="e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73" Nov 02 03:29:31.173345 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:31.173311 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73\": container with ID starting with e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73 not found: ID does not exist" containerID="e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73" Nov 02 03:29:31.173507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.173477 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73"} err="failed to get container status \"e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73\": rpc error: code = NotFound desc = could not find container \"e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73\": container with ID starting with e3634b65ae3c69fc7ba1466547fec3978905d53526bdae908bc738a9f1429d73 not found: ID does not exist" Nov 02 03:29:31.325234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.325205 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc"] Nov 02 03:29:31.374559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:31.374529 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-xndmc"] Nov 02 03:29:32.114465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:32.114433 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" path="/var/lib/kubelet/pods/03e7f592-6cca-4fff-8711-ea3f7f39a578/volumes" Nov 02 03:29:33.445520 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:33.445481 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:29:36.262845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.262815 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x"] Nov 02 03:29:36.263313 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.263300 4868 cpu_manager.go:414] "containerMap: removing container" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" Nov 02 03:29:36.263372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.263314 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" Nov 02 03:29:36.263411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.263404 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e7f592-6cca-4fff-8711-ea3f7f39a578" containerName="istio-proxy" Nov 02 03:29:36.266546 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.266531 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.348217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.348166 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.348733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.348697 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7m55\" (UniqueName: \"kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55\") pod \"payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.374403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.368782 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d94cc3\"" Nov 02 03:29:36.374403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.372526 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d94cc3-dockercfg-bj6jz\"" Nov 02 03:29:36.374403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.373187 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x"] Nov 02 03:29:36.450325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.450281 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7m55\" (UniqueName: \"kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55\") pod \"payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.450558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.450445 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.451713 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.451674 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.507146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.507115 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7m55\" (UniqueName: \"kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55\") pod \"payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.579030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.578956 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:36.950040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:36.950009 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x"] Nov 02 03:29:36.951909 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:29:36.951881 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9445ad_44a7_43a1_948a_d14b7363d153.slice/crio-555f55e412299e3b25d2fe21b7e9d622105eaf88679ff1114cedaf412d1652af WatchSource:0}: Error finding container 555f55e412299e3b25d2fe21b7e9d622105eaf88679ff1114cedaf412d1652af: Status 404 returned error can't find the container with id 555f55e412299e3b25d2fe21b7e9d622105eaf88679ff1114cedaf412d1652af Nov 02 03:29:37.200126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:37.200042 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" event={"ID":"7a9445ad-44a7-43a1-948a-d14b7363d153","Type":"ContainerStarted","Data":"1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705"} Nov 02 03:29:37.200126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:37.200100 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" event={"ID":"7a9445ad-44a7-43a1-948a-d14b7363d153","Type":"ContainerStarted","Data":"555f55e412299e3b25d2fe21b7e9d622105eaf88679ff1114cedaf412d1652af"} Nov 02 03:29:37.200342 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:37.200151 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:37.312718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:37.312668 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" podStartSLOduration=1.312656094 podStartE2EDuration="1.312656094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:37.311324091 +0000 UTC m=+2519.307025033" watchObservedRunningTime="2026-11-02 03:29:37.312656094 +0000 UTC m=+2519.308357036" Nov 02 03:29:40.531294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.531258 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb"] Nov 02 03:29:40.536499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.536480 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.588899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.588862 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f1a692-serving-cert\"" Nov 02 03:29:40.591989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.591961 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb"] Nov 02 03:29:40.611491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.611466 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.611591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.611509 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.611696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.611673 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2rg\" (UniqueName: \"kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.611737 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.611714 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.712480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.712447 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.712609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.712523 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.712609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.712561 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.712718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.712702 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2rg\" (UniqueName: \"kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.712787 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:40.712763 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-serving-cert: secret "maas-api-e2e-api-b-f1a692-serving-cert" not found Nov 02 03:29:40.712855 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:29:40.712843 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls podName:c4bf38d8-4c58-4b5d-87e5-0c1432af7f71 nodeName:}" failed. No retries permitted until 2026-11-02 03:29:41.212823092 +0000 UTC m=+2523.208524033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls") pod "maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" (UID: "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71") : secret "maas-api-e2e-api-b-f1a692-serving-cert" not found Nov 02 03:29:40.713134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.713111 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.714989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.714969 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:40.769647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:40.769614 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2rg\" (UniqueName: \"kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:41.218276 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:41.218245 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:41.220787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:41.220757 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\") pod \"maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:41.450161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:41.450129 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:41.820691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:41.820661 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb"] Nov 02 03:29:41.822819 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:29:41.822793 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bf38d8_4c58_4b5d_87e5_0c1432af7f71.slice/crio-91fa3af300bcb15a78a6c95fa2d00219c78e4e09db3776ce2e8c231c4b5f20a0 WatchSource:0}: Error finding container 91fa3af300bcb15a78a6c95fa2d00219c78e4e09db3776ce2e8c231c4b5f20a0: Status 404 returned error can't find the container with id 91fa3af300bcb15a78a6c95fa2d00219c78e4e09db3776ce2e8c231c4b5f20a0 Nov 02 03:29:42.229181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:42.229146 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" event={"ID":"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71","Type":"ContainerStarted","Data":"91fa3af300bcb15a78a6c95fa2d00219c78e4e09db3776ce2e8c231c4b5f20a0"} Nov 02 03:29:43.287375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:43.286887 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" event={"ID":"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71","Type":"ContainerStarted","Data":"28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe"} Nov 02 03:29:43.295388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:43.287910 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:29:43.409387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:43.409321 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" podStartSLOduration=2.551248598 podStartE2EDuration="3.409305771s" totalImagesPullingTime="858.057173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:29:43.407910812 +0000 UTC m=+2525.403611755" watchObservedRunningTime="2026-11-02 03:29:43.409305771 +0000 UTC m=+2525.405006744" Nov 02 03:29:48.209154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:48.209128 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 03:29:50.329082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:29:50.329054 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:30:00.179859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.179821 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv"] Nov 02 03:30:00.184091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.184069 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6"] Nov 02 03:30:00.184216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.184177 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.187687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.187672 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt"] Nov 02 03:30:00.187809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.187789 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.191189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.191173 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.229171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.229141 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562"] Nov 02 03:30:00.232844 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.232824 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.246415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.246390 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.246528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.246423 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mtb\" (UniqueName: \"kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.246653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.246568 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.246771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.246755 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.246812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.246790 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzml\" (UniqueName: \"kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml\") pod \"maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.246847 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.246812 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdzg\" (UniqueName: \"kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.264605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.264577 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv"] Nov 02 03:30:00.270210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.270180 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt"] Nov 02 03:30:00.272928 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.272896 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6"] Nov 02 03:30:00.319089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.319066 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:30:00.335558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.335526 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562"] Nov 02 03:30:00.347495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347467 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.347639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347503 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzml\" (UniqueName: \"kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml\") pod \"maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.347639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347524 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdzg\" (UniqueName: \"kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.347639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347558 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.347639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347585 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mtb\" (UniqueName: \"kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.348284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347680 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.348284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347812 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.348284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.347838 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgpq\" (UniqueName: \"kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq\") pod \"maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.348284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.348181 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.348525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.348407 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.348525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.348412 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.403847 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.403817 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdzg\" (UniqueName: \"kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.403978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.403895 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzml\" (UniqueName: \"kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml\") pod \"maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.403978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.403931 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mtb\" (UniqueName: \"kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.448925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.448864 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.448925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.448915 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgpq\" (UniqueName: \"kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq\") pod \"maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.449499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.449481 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.499476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.499455 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:00.505884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.505863 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgpq\" (UniqueName: \"kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq\") pod \"maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.507930 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.507907 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:00.514574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.514552 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:00.546406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.546370 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:00.886787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.886759 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv"] Nov 02 03:30:00.887673 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:30:00.887650 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11ac0e2_281c_414f_a001_b6d052f63c0b.slice/crio-da7105fff8decbf1772212c06a6d41d1f93d89b1c2e71e9132a4d0a197645e7a WatchSource:0}: Error finding container da7105fff8decbf1772212c06a6d41d1f93d89b1c2e71e9132a4d0a197645e7a: Status 404 returned error can't find the container with id da7105fff8decbf1772212c06a6d41d1f93d89b1c2e71e9132a4d0a197645e7a Nov 02 03:30:00.933638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.933610 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt"] Nov 02 03:30:00.933768 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:30:00.933747 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38feafd2_fd66_401f_bd55_eb61ab7219cc.slice/crio-fbc308d796b411b0b94f2cb5451f012230f5496ca1364c457044b6dfc870b404 WatchSource:0}: Error finding container fbc308d796b411b0b94f2cb5451f012230f5496ca1364c457044b6dfc870b404: Status 404 returned error can't find the container with id fbc308d796b411b0b94f2cb5451f012230f5496ca1364c457044b6dfc870b404 Nov 02 03:30:00.936486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.936465 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6"] Nov 02 03:30:00.937938 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:30:00.937913 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c830657_7524_4526_9cb6_18fac3c5f710.slice/crio-d74566430bfdcf8bf5e0b4acc55a0ca674687eaba2a68ca11662788c2021e2dd WatchSource:0}: Error finding container d74566430bfdcf8bf5e0b4acc55a0ca674687eaba2a68ca11662788c2021e2dd: Status 404 returned error can't find the container with id d74566430bfdcf8bf5e0b4acc55a0ca674687eaba2a68ca11662788c2021e2dd Nov 02 03:30:00.983581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:00.983560 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562"] Nov 02 03:30:00.985152 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:30:00.985130 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5535233f_d1b6_433f_93ce_5f6a48cfb0ac.slice/crio-bdfd086870bba213395abccb3a1c3bc74b5f8578ad2551d2541902a7fc0046d3 WatchSource:0}: Error finding container bdfd086870bba213395abccb3a1c3bc74b5f8578ad2551d2541902a7fc0046d3: Status 404 returned error can't find the container with id bdfd086870bba213395abccb3a1c3bc74b5f8578ad2551d2541902a7fc0046d3 Nov 02 03:30:01.419390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.419288 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerStarted","Data":"9f4117d75204547328644d2abb27101465736bb83bd40bb672f3778f1fffbdf7"} Nov 02 03:30:01.419390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.419375 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerStarted","Data":"bdfd086870bba213395abccb3a1c3bc74b5f8578ad2551d2541902a7fc0046d3"} Nov 02 03:30:01.421294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.421265 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerStarted","Data":"e16405c155c054437eb388f7c14de290daa25e4d3d3d82e31288f2125ba346aa"} Nov 02 03:30:01.421462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.421331 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerStarted","Data":"fbc308d796b411b0b94f2cb5451f012230f5496ca1364c457044b6dfc870b404"} Nov 02 03:30:01.423087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.423062 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerStarted","Data":"c867551dacc7af325c8addc718ade47d4be8a1062f80affeeb70043c4973ac29"} Nov 02 03:30:01.423217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.423107 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerStarted","Data":"da7105fff8decbf1772212c06a6d41d1f93d89b1c2e71e9132a4d0a197645e7a"} Nov 02 03:30:01.424660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.424641 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerStarted","Data":"e3d20654c28b1ccaf00747b36afbe2130c724451d7533f4875c3600dcf60d6d5"} Nov 02 03:30:01.424762 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.424675 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerStarted","Data":"d74566430bfdcf8bf5e0b4acc55a0ca674687eaba2a68ca11662788c2021e2dd"} Nov 02 03:30:01.530245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.530185 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" podStartSLOduration=1.530168664 podStartE2EDuration="1.530168664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:01.529881356 +0000 UTC m=+2543.525582299" watchObservedRunningTime="2026-11-02 03:30:01.530168664 +0000 UTC m=+2543.525869608" Nov 02 03:30:01.641096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.641037 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" podStartSLOduration=1.6410205919999998 podStartE2EDuration="1.641020592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:01.640736491 +0000 UTC m=+2543.636437460" watchObservedRunningTime="2026-11-02 03:30:01.641020592 +0000 UTC m=+2543.636721537" Nov 02 03:30:01.755580 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:01.755522 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" podStartSLOduration=1.755502326 podStartE2EDuration="1.755502326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:01.752491797 +0000 UTC m=+2543.748192739" watchObservedRunningTime="2026-11-02 03:30:01.755502326 +0000 UTC m=+2543.751203284" Nov 02 03:30:02.431868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.431829 4868 generic.go:356] "Generic (PLEG): container finished" podID="0c830657-7524-4526-9cb6-18fac3c5f710" containerID="e3d20654c28b1ccaf00747b36afbe2130c724451d7533f4875c3600dcf60d6d5" exitCode=60 Nov 02 03:30:02.432340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.431914 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerDied","Data":"e3d20654c28b1ccaf00747b36afbe2130c724451d7533f4875c3600dcf60d6d5"} Nov 02 03:30:02.432340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.432185 4868 scope.go:122] "RemoveContainer" containerID="e3d20654c28b1ccaf00747b36afbe2130c724451d7533f4875c3600dcf60d6d5" Nov 02 03:30:02.433680 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.433659 4868 generic.go:356] "Generic (PLEG): container finished" podID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerID="9f4117d75204547328644d2abb27101465736bb83bd40bb672f3778f1fffbdf7" exitCode=60 Nov 02 03:30:02.433829 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.433752 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerDied","Data":"9f4117d75204547328644d2abb27101465736bb83bd40bb672f3778f1fffbdf7"} Nov 02 03:30:02.433974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.433952 4868 scope.go:122] "RemoveContainer" containerID="9f4117d75204547328644d2abb27101465736bb83bd40bb672f3778f1fffbdf7" Nov 02 03:30:02.435558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.435424 4868 generic.go:356] "Generic (PLEG): container finished" podID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerID="e16405c155c054437eb388f7c14de290daa25e4d3d3d82e31288f2125ba346aa" exitCode=60 Nov 02 03:30:02.435558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.435509 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerDied","Data":"e16405c155c054437eb388f7c14de290daa25e4d3d3d82e31288f2125ba346aa"} Nov 02 03:30:02.435731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.435711 4868 scope.go:122] "RemoveContainer" containerID="e16405c155c054437eb388f7c14de290daa25e4d3d3d82e31288f2125ba346aa" Nov 02 03:30:02.437489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.437470 4868 generic.go:356] "Generic (PLEG): container finished" podID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerID="c867551dacc7af325c8addc718ade47d4be8a1062f80affeeb70043c4973ac29" exitCode=60 Nov 02 03:30:02.437584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.437553 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerDied","Data":"c867551dacc7af325c8addc718ade47d4be8a1062f80affeeb70043c4973ac29"} Nov 02 03:30:02.437709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:02.437698 4868 scope.go:122] "RemoveContainer" containerID="c867551dacc7af325c8addc718ade47d4be8a1062f80affeeb70043c4973ac29" Nov 02 03:30:03.447422 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.447388 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerStarted","Data":"50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518"} Nov 02 03:30:03.448198 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:03.448175 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:30:03.449336 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.449309 4868 generic.go:356] "Generic (PLEG): container finished" podID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerID="e5de1d60127ddfe17191447c07923b69aef8f053ae79239acb6413fd50f52a70" exitCode=60 Nov 02 03:30:03.449451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.449380 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerDied","Data":"e5de1d60127ddfe17191447c07923b69aef8f053ae79239acb6413fd50f52a70"} Nov 02 03:30:03.449451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.449433 4868 scope.go:122] "RemoveContainer" containerID="9f4117d75204547328644d2abb27101465736bb83bd40bb672f3778f1fffbdf7" Nov 02 03:30:03.449682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.449657 4868 scope.go:122] "RemoveContainer" containerID="e5de1d60127ddfe17191447c07923b69aef8f053ae79239acb6413fd50f52a70" Nov 02 03:30:03.450027 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:03.449938 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562_odh-ai-gateway-infra(5535233f-d1b6-433f-93ce-5f6a48cfb0ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" Nov 02 03:30:03.451883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.451863 4868 generic.go:356] "Generic (PLEG): container finished" podID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerID="cef93ce4c83175a8dd0e46f728397431c94e614fede11949702372e91074d805" exitCode=60 Nov 02 03:30:03.451965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.451944 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerDied","Data":"cef93ce4c83175a8dd0e46f728397431c94e614fede11949702372e91074d805"} Nov 02 03:30:03.452135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.452121 4868 scope.go:122] "RemoveContainer" containerID="cef93ce4c83175a8dd0e46f728397431c94e614fede11949702372e91074d805" Nov 02 03:30:03.452331 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:03.452313 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt_odh-ai-gateway-infra(38feafd2-fd66-401f-bd55-eb61ab7219cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" Nov 02 03:30:03.453633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.453608 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerStarted","Data":"224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4"} Nov 02 03:30:03.465428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.465402 4868 scope.go:122] "RemoveContainer" containerID="e16405c155c054437eb388f7c14de290daa25e4d3d3d82e31288f2125ba346aa" Nov 02 03:30:03.564155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:03.564115 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" podStartSLOduration=3.564101909 podStartE2EDuration="3.564101909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:01.869702477 +0000 UTC m=+2543.865403424" watchObservedRunningTime="2026-11-02 03:30:03.564101909 +0000 UTC m=+2545.559802849" Nov 02 03:30:04.461216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.461180 4868 generic.go:356] "Generic (PLEG): container finished" podID="0c830657-7524-4526-9cb6-18fac3c5f710" containerID="50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518" exitCode=60 Nov 02 03:30:04.461626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.461261 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerDied","Data":"50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518"} Nov 02 03:30:04.461626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.461314 4868 scope.go:122] "RemoveContainer" containerID="e3d20654c28b1ccaf00747b36afbe2130c724451d7533f4875c3600dcf60d6d5" Nov 02 03:30:04.461626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.461523 4868 scope.go:122] "RemoveContainer" containerID="50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518" Nov 02 03:30:04.461849 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:04.461819 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6_odh-ai-gateway-infra(0c830657-7524-4526-9cb6-18fac3c5f710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" Nov 02 03:30:04.463327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.463307 4868 scope.go:122] "RemoveContainer" containerID="e5de1d60127ddfe17191447c07923b69aef8f053ae79239acb6413fd50f52a70" Nov 02 03:30:04.463622 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:04.463595 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562_odh-ai-gateway-infra(5535233f-d1b6-433f-93ce-5f6a48cfb0ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" Nov 02 03:30:04.464814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.464800 4868 scope.go:122] "RemoveContainer" containerID="cef93ce4c83175a8dd0e46f728397431c94e614fede11949702372e91074d805" Nov 02 03:30:04.465008 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:04.464987 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt_odh-ai-gateway-infra(38feafd2-fd66-401f-bd55-eb61ab7219cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" Nov 02 03:30:04.466450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.466425 4868 generic.go:356] "Generic (PLEG): container finished" podID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerID="224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4" exitCode=60 Nov 02 03:30:04.466540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.466458 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerDied","Data":"224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4"} Nov 02 03:30:04.466726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.466709 4868 scope.go:122] "RemoveContainer" containerID="224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4" Nov 02 03:30:04.466972 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:04.466953 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv_odh-ai-gateway-infra(e11ac0e2-281c-414f-a001-b6d052f63c0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" Nov 02 03:30:04.474530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:04.474507 4868 scope.go:122] "RemoveContainer" containerID="c867551dacc7af325c8addc718ade47d4be8a1062f80affeeb70043c4973ac29" Nov 02 03:30:05.475031 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:05.474997 4868 scope.go:122] "RemoveContainer" containerID="224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4" Nov 02 03:30:05.475496 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:05.475282 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv_odh-ai-gateway-infra(e11ac0e2-281c-414f-a001-b6d052f63c0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" Nov 02 03:30:05.476595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:05.476568 4868 scope.go:122] "RemoveContainer" containerID="50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518" Nov 02 03:30:05.476806 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:05.476788 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6_odh-ai-gateway-infra(0c830657-7524-4526-9cb6-18fac3c5f710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" Nov 02 03:30:14.108488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:14.108455 4868 scope.go:122] "RemoveContainer" containerID="cef93ce4c83175a8dd0e46f728397431c94e614fede11949702372e91074d805" Nov 02 03:30:14.532698 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:14.532671 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerStarted","Data":"48566ef44db26cfe8424899f299a6d7d20edfe456fced35b406f32893eb086bd"} Nov 02 03:30:15.585375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:15.577319 4868 generic.go:356] "Generic (PLEG): container finished" podID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerID="48566ef44db26cfe8424899f299a6d7d20edfe456fced35b406f32893eb086bd" exitCode=60 Nov 02 03:30:15.585375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:15.577383 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerDied","Data":"48566ef44db26cfe8424899f299a6d7d20edfe456fced35b406f32893eb086bd"} Nov 02 03:30:15.585375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:15.577440 4868 scope.go:122] "RemoveContainer" containerID="cef93ce4c83175a8dd0e46f728397431c94e614fede11949702372e91074d805" Nov 02 03:30:15.585375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:15.577908 4868 scope.go:122] "RemoveContainer" containerID="48566ef44db26cfe8424899f299a6d7d20edfe456fced35b406f32893eb086bd" Nov 02 03:30:15.585375 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:15.578324 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt_odh-ai-gateway-infra(38feafd2-fd66-401f-bd55-eb61ab7219cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" Nov 02 03:30:15.682763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:15.682732 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt"] Nov 02 03:30:16.108727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.108696 4868 scope.go:122] "RemoveContainer" containerID="50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518" Nov 02 03:30:16.584766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.584737 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerStarted","Data":"321a0dd056981f75d00ad4523d31b152a7319974f7de671b8f35214d63527e25"} Nov 02 03:30:16.731115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.731090 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:16.845611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.845583 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg\" (UniqueName: \"kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg\") pod \"38feafd2-fd66-401f-bd55-eb61ab7219cc\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " Nov 02 03:30:16.845783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.845681 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca\") pod \"38feafd2-fd66-401f-bd55-eb61ab7219cc\" (UID: \"38feafd2-fd66-401f-bd55-eb61ab7219cc\") " Nov 02 03:30:16.846009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.845986 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca" pod "38feafd2-fd66-401f-bd55-eb61ab7219cc" (UID: "38feafd2-fd66-401f-bd55-eb61ab7219cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:30:16.847711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.847689 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg" pod "38feafd2-fd66-401f-bd55-eb61ab7219cc" (UID: "38feafd2-fd66-401f-bd55-eb61ab7219cc"). InnerVolumeSpecName "kube-api-access-tbdzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:30:16.947561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.947488 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38feafd2-fd66-401f-bd55-eb61ab7219cc-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:16.947561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:16.947525 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbdzg\" (UniqueName: \"kubernetes.io/projected/38feafd2-fd66-401f-bd55-eb61ab7219cc-kube-api-access-tbdzg\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:17.107857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.107830 4868 scope.go:122] "RemoveContainer" containerID="e5de1d60127ddfe17191447c07923b69aef8f053ae79239acb6413fd50f52a70" Nov 02 03:30:17.628374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.624575 4868 generic.go:356] "Generic (PLEG): container finished" podID="0c830657-7524-4526-9cb6-18fac3c5f710" containerID="321a0dd056981f75d00ad4523d31b152a7319974f7de671b8f35214d63527e25" exitCode=60 Nov 02 03:30:17.628374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.624723 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerDied","Data":"321a0dd056981f75d00ad4523d31b152a7319974f7de671b8f35214d63527e25"} Nov 02 03:30:17.628374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.624813 4868 scope.go:122] "RemoveContainer" containerID="50ca1ef157533c1229e37a5969cb753424343fa114bb45e6c82877e98844c518" Nov 02 03:30:17.628374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.625266 4868 scope.go:122] "RemoveContainer" containerID="321a0dd056981f75d00ad4523d31b152a7319974f7de671b8f35214d63527e25" Nov 02 03:30:17.628374 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:17.625687 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6_odh-ai-gateway-infra(0c830657-7524-4526-9cb6-18fac3c5f710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" Nov 02 03:30:17.645084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.645023 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerStarted","Data":"b3bb8517542d7e9679aeb482f9907b99460472394994356a9dee7aa231b8314c"} Nov 02 03:30:17.655370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.648309 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" event={"ID":"38feafd2-fd66-401f-bd55-eb61ab7219cc","Type":"ContainerDied","Data":"fbc308d796b411b0b94f2cb5451f012230f5496ca1364c457044b6dfc870b404"} Nov 02 03:30:17.655370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.648418 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt" Nov 02 03:30:17.702568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.702540 4868 scope.go:122] "RemoveContainer" containerID="48566ef44db26cfe8424899f299a6d7d20edfe456fced35b406f32893eb086bd" Nov 02 03:30:17.731862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.731839 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6"] Nov 02 03:30:17.980094 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.980065 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt"] Nov 02 03:30:17.988604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:17.988571 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893170-mr9tt"] Nov 02 03:30:18.111329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.111302 4868 scope.go:122] "RemoveContainer" containerID="224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4" Nov 02 03:30:18.116594 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.116570 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" path="/var/lib/kubelet/pods/38feafd2-fd66-401f-bd55-eb61ab7219cc/volumes" Nov 02 03:30:18.656799 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.656765 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerStarted","Data":"58468d9d6e24aff8152609c20a14eb79bdd5411d7bf99e781fc4d3c377755c9a"} Nov 02 03:30:18.660381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.660341 4868 generic.go:356] "Generic (PLEG): container finished" podID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerID="b3bb8517542d7e9679aeb482f9907b99460472394994356a9dee7aa231b8314c" exitCode=60 Nov 02 03:30:18.660524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.660416 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerDied","Data":"b3bb8517542d7e9679aeb482f9907b99460472394994356a9dee7aa231b8314c"} Nov 02 03:30:18.660524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.660473 4868 scope.go:122] "RemoveContainer" containerID="e5de1d60127ddfe17191447c07923b69aef8f053ae79239acb6413fd50f52a70" Nov 02 03:30:18.660917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.660892 4868 scope.go:122] "RemoveContainer" containerID="b3bb8517542d7e9679aeb482f9907b99460472394994356a9dee7aa231b8314c" Nov 02 03:30:18.661250 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:18.661227 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562_odh-ai-gateway-infra(5535233f-d1b6-433f-93ce-5f6a48cfb0ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" Nov 02 03:30:18.879333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.879246 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562"] Nov 02 03:30:18.924879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.924812 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:18.974040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.974012 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb\" (UniqueName: \"kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb\") pod \"0c830657-7524-4526-9cb6-18fac3c5f710\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " Nov 02 03:30:18.974173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.974093 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca\") pod \"0c830657-7524-4526-9cb6-18fac3c5f710\" (UID: \"0c830657-7524-4526-9cb6-18fac3c5f710\") " Nov 02 03:30:18.974505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.974476 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca" pod "0c830657-7524-4526-9cb6-18fac3c5f710" (UID: "0c830657-7524-4526-9cb6-18fac3c5f710"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:30:18.976023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:18.976001 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb" pod "0c830657-7524-4526-9cb6-18fac3c5f710" (UID: "0c830657-7524-4526-9cb6-18fac3c5f710"). InnerVolumeSpecName "kube-api-access-w4mtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:30:19.075230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.075201 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4mtb\" (UniqueName: \"kubernetes.io/projected/0c830657-7524-4526-9cb6-18fac3c5f710-kube-api-access-w4mtb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:19.075230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.075227 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c830657-7524-4526-9cb6-18fac3c5f710-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:19.705046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.705012 4868 generic.go:356] "Generic (PLEG): container finished" podID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerID="58468d9d6e24aff8152609c20a14eb79bdd5411d7bf99e781fc4d3c377755c9a" exitCode=60 Nov 02 03:30:19.705263 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.705083 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerDied","Data":"58468d9d6e24aff8152609c20a14eb79bdd5411d7bf99e781fc4d3c377755c9a"} Nov 02 03:30:19.705263 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.705148 4868 scope.go:122] "RemoveContainer" containerID="224dee1e915cb777cb063beee27748f71cf3bf998458b32d824bfa80182fd1e4" Nov 02 03:30:19.705563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.705548 4868 scope.go:122] "RemoveContainer" containerID="58468d9d6e24aff8152609c20a14eb79bdd5411d7bf99e781fc4d3c377755c9a" Nov 02 03:30:19.705855 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:19.705836 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv_odh-ai-gateway-infra(e11ac0e2-281c-414f-a001-b6d052f63c0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" Nov 02 03:30:19.708516 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.708499 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" Nov 02 03:30:19.708597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.708547 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6" event={"ID":"0c830657-7524-4526-9cb6-18fac3c5f710","Type":"ContainerDied","Data":"d74566430bfdcf8bf5e0b4acc55a0ca674687eaba2a68ca11662788c2021e2dd"} Nov 02 03:30:19.721315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.721301 4868 scope.go:122] "RemoveContainer" containerID="321a0dd056981f75d00ad4523d31b152a7319974f7de671b8f35214d63527e25" Nov 02 03:30:19.811734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:19.811691 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv"] Nov 02 03:30:20.011407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.011341 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:20.085618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.085577 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6"] Nov 02 03:30:20.087908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.087881 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893170-mmsq6"] Nov 02 03:30:20.099977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.099948 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq\" (UniqueName: \"kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq\") pod \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " Nov 02 03:30:20.100098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.100071 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca\") pod \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\" (UID: \"5535233f-d1b6-433f-93ce-5f6a48cfb0ac\") " Nov 02 03:30:20.100667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.100627 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca" pod "5535233f-d1b6-433f-93ce-5f6a48cfb0ac" (UID: "5535233f-d1b6-433f-93ce-5f6a48cfb0ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:30:20.102011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.101990 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq" pod "5535233f-d1b6-433f-93ce-5f6a48cfb0ac" (UID: "5535233f-d1b6-433f-93ce-5f6a48cfb0ac"). InnerVolumeSpecName "kube-api-access-szgpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:30:20.116359 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.116326 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" path="/var/lib/kubelet/pods/0c830657-7524-4526-9cb6-18fac3c5f710/volumes" Nov 02 03:30:20.202058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.202025 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:20.202058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.202059 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgpq\" (UniqueName: \"kubernetes.io/projected/5535233f-d1b6-433f-93ce-5f6a48cfb0ac-kube-api-access-szgpq\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:20.991371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.991135 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78"] Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.991998 4868 cpu_manager.go:414] "containerMap: removing container" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992024 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992051 4868 cpu_manager.go:414] "containerMap: removing container" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992058 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992072 4868 cpu_manager.go:414] "containerMap: removing container" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992080 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992102 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992109 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992124 4868 cpu_manager.go:414] "containerMap: removing container" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992130 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992151 4868 cpu_manager.go:414] "containerMap: removing container" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992157 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992172 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992178 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992193 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992200 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992382 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992398 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992413 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992423 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992433 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c830657-7524-4526-9cb6-18fac3c5f710" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992444 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:20.992453 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:30:21.008997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.003884 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.030971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.030071 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" event={"ID":"5535233f-d1b6-433f-93ce-5f6a48cfb0ac","Type":"ContainerDied","Data":"bdfd086870bba213395abccb3a1c3bc74b5f8578ad2551d2541902a7fc0046d3"} Nov 02 03:30:21.030971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.030147 4868 scope.go:122] "RemoveContainer" containerID="b3bb8517542d7e9679aeb482f9907b99460472394994356a9dee7aa231b8314c" Nov 02 03:30:21.030971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.030297 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056549 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnk5\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056636 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056670 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056723 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056754 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056875 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056928 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056959 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.056983 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.057436 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d5625-openshift-default-dockercfg-nf9dz\"" Nov 02 03:30:21.096769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.094486 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78"] Nov 02 03:30:21.175018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.163605 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.175018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.166504 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.175018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.166584 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.175018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.166638 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.189433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.188971 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.189433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.189345 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.208757 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.208855 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.209192 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.215152 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.223454 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.223514 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.223838 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.224474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.223996 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnk5\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.326470 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.288710 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.356407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.330619 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnk5\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.422312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.422225 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.444375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.440779 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo\") pod \"e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:21.444375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.442840 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562"] Nov 02 03:30:21.495383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.492133 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dd87f0-29893170-xn562"] Nov 02 03:30:21.889091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:21.888997 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:22.105850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.105528 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:22.224451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.214239 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca\") pod \"e11ac0e2-281c-414f-a001-b6d052f63c0b\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " Nov 02 03:30:22.224451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.214325 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml\" (UniqueName: \"kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml\") pod \"e11ac0e2-281c-414f-a001-b6d052f63c0b\" (UID: \"e11ac0e2-281c-414f-a001-b6d052f63c0b\") " Nov 02 03:30:22.257401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.233526 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca" pod "e11ac0e2-281c-414f-a001-b6d052f63c0b" (UID: "e11ac0e2-281c-414f-a001-b6d052f63c0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:30:22.259435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.257791 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml" pod "e11ac0e2-281c-414f-a001-b6d052f63c0b" (UID: "e11ac0e2-281c-414f-a001-b6d052f63c0b"). InnerVolumeSpecName "kube-api-access-rxzml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:30:22.319574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.306557 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" path="/var/lib/kubelet/pods/5535233f-d1b6-433f-93ce-5f6a48cfb0ac/volumes" Nov 02 03:30:22.319574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.318204 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11ac0e2-281c-414f-a001-b6d052f63c0b-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:22.319574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.318259 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzml\" (UniqueName: \"kubernetes.io/projected/e11ac0e2-281c-414f-a001-b6d052f63c0b-kube-api-access-rxzml\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:30:22.425882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.424689 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" event={"ID":"e11ac0e2-281c-414f-a001-b6d052f63c0b","Type":"ContainerDied","Data":"da7105fff8decbf1772212c06a6d41d1f93d89b1c2e71e9132a4d0a197645e7a"} Nov 02 03:30:22.425882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.424766 4868 scope.go:122] "RemoveContainer" containerID="58468d9d6e24aff8152609c20a14eb79bdd5411d7bf99e781fc4d3c377755c9a" Nov 02 03:30:22.425882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.424907 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv" Nov 02 03:30:22.628185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.628148 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv"] Nov 02 03:30:22.664019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.663981 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-045653-29893170-9p9mv"] Nov 02 03:30:22.895096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.895073 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78"] Nov 02 03:30:22.897584 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:30:22.897556 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5cac0b_7b7c_4b11_aafb_bf00572a2ba4.slice/crio-1c3e346c02d51ee3840d11424376b05305b0c695fb4627940d45587f9ea7b9d4 WatchSource:0}: Error finding container 1c3e346c02d51ee3840d11424376b05305b0c695fb4627940d45587f9ea7b9d4: Status 404 returned error can't find the container with id 1c3e346c02d51ee3840d11424376b05305b0c695fb4627940d45587f9ea7b9d4 Nov 02 03:30:22.900133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.900106 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:22.900191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.900165 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:22.900230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:22.900197 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:23.442547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:23.442496 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" event={"ID":"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4","Type":"ContainerStarted","Data":"bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa"} Nov 02 03:30:23.443183 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:23.443123 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" event={"ID":"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4","Type":"ContainerStarted","Data":"1c3e346c02d51ee3840d11424376b05305b0c695fb4627940d45587f9ea7b9d4"} Nov 02 03:30:23.580377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:23.571195 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podStartSLOduration=3.571175806 podStartE2EDuration="3.571175806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:30:23.55916984 +0000 UTC m=+2565.554870796" watchObservedRunningTime="2026-11-02 03:30:23.571175806 +0000 UTC m=+2565.566876749" Nov 02 03:30:23.893376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:23.891439 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:24.116708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:24.116673 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" path="/var/lib/kubelet/pods/e11ac0e2-281c-414f-a001-b6d052f63c0b/volumes" Nov 02 03:30:24.896642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:24.896613 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:25.482728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:25.482695 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:25.483695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:25.483671 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:30:29.511242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.511209 4868 generic.go:356] "Generic (PLEG): container finished" podID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerID="f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8" exitCode=0 Nov 02 03:30:29.511821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.511281 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" event={"ID":"a9a68900-05c8-4e9c-b89d-ef751561e4f7","Type":"ContainerDied","Data":"f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8"} Nov 02 03:30:29.511821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.511726 4868 scope.go:122] "RemoveContainer" containerID="f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8" Nov 02 03:30:29.513208 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.513185 4868 generic.go:356] "Generic (PLEG): container finished" podID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerID="2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478" exitCode=0 Nov 02 03:30:29.513296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.513255 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" event={"ID":"f7ecbe05-9f48-4b2a-8654-0823907a23aa","Type":"ContainerDied","Data":"2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478"} Nov 02 03:30:29.513568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.513538 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:29.513616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.513567 4868 scope.go:122] "RemoveContainer" containerID="2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478" Nov 02 03:30:29.513654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.513622 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:29.513687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.513669 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:29.514888 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.514860 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:29.514982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.514926 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:29.514982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:29.514961 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:30:30.521158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.521120 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" event={"ID":"a9a68900-05c8-4e9c-b89d-ef751561e4f7","Type":"ContainerStarted","Data":"ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5"} Nov 02 03:30:30.523130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.523103 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" event={"ID":"f7ecbe05-9f48-4b2a-8654-0823907a23aa","Type":"ContainerStarted","Data":"f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68"} Nov 02 03:30:30.659939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.659904 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:30:30.660284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.660260 4868 patch_prober.go:29] interesting pod/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 03:30:30.660597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.660496 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 03:30:30.675933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.675903 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:30:30.676287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.676259 4868 patch_prober.go:29] interesting pod/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 03:30:30.676503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.676477 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 03:30:30.787434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:30.777645 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:30:31.661377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:31.660732 4868 patch_prober.go:29] interesting pod/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 03:30:31.661377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:31.660796 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 03:30:31.681453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:31.678737 4868 patch_prober.go:29] interesting pod/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 03:30:31.681453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:31.678812 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 03:30:32.650380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:32.648695 4868 patch_prober.go:29] interesting pod/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 03:30:32.650380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:32.648758 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 03:30:32.674374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:32.673609 4868 patch_prober.go:29] interesting pod/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 03:30:32.674374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:32.673675 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 03:30:33.490076 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:30:33.490040 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:30:33.653858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:33.653820 4868 patch_prober.go:29] interesting pod/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 03:30:33.654147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:33.654121 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 03:30:33.676602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:33.676565 4868 patch_prober.go:29] interesting pod/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 03:30:33.677193 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:33.677161 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 03:30:34.654376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:34.650387 4868 patch_prober.go:29] interesting pod/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 03:30:34.654376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:34.650451 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 03:30:34.674716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:34.674684 4868 patch_prober.go:29] interesting pod/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 03:30:34.674980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:34.674948 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 03:30:35.651372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:35.648758 4868 patch_prober.go:29] interesting pod/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 03:30:35.651372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:35.648820 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 03:30:35.675784 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:35.675752 4868 patch_prober.go:29] interesting pod/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 03:30:35.676050 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:35.676013 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 03:30:35.864380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:35.863433 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:30:36.652155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:36.652126 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:30:36.653141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:36.653118 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:30:36.677537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:36.677513 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:30:36.678443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:30:36.678427 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:31:03.446411 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:31:03.446375 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:31:12.310317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.310279 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf"] Nov 02 03:31:12.311069 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311044 4868 cpu_manager.go:414] "containerMap: removing container" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:31:12.311122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311073 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:31:12.311122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311107 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311116 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311131 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311139 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311172 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311179 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311320 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="38feafd2-fd66-401f-bd55-eb61ab7219cc" containerName="cleanup" Nov 02 03:31:12.311335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311333 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311344 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.311415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311369 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="5535233f-d1b6-433f-93ce-5f6a48cfb0ac" containerName="cleanup" Nov 02 03:31:12.311415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.311381 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11ac0e2-281c-414f-a001-b6d052f63c0b" containerName="cleanup" Nov 02 03:31:12.315031 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.315012 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.394606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.394568 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0809c8-openshift-default-dockercfg-57tsl\"" Nov 02 03:31:12.432997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.432956 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf"] Nov 02 03:31:12.473733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.464102 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.474177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.474064 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.474513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.474488 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.474733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.474718 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.474891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.474874 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.475064 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.475048 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.475255 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.475241 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.475481 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.475464 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvv9r\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.475647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.475633 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.577288 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.577197 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.578068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.578045 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.581343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.581316 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.582013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.581991 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.582708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.582687 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvv9r\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.583153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.583135 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.583643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.583626 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.584049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.584030 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.600741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.600669 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.601731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.583524 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.601902 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.582540 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.602009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.577970 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.602097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.599698 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.602189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.600280 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.602286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.578451 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.630080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.630042 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.642386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.642343 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.644279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.644253 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvv9r\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r\") pod \"e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:12.651985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:12.651962 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:13.087399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:13.087369 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf"] Nov 02 03:31:13.091952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:13.090574 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:31:13.091952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:13.090672 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:31:13.091952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:13.090719 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:31:13.832907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:13.832864 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" event={"ID":"e6857c35-deec-4921-867b-e4e21ee15205","Type":"ContainerStarted","Data":"2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f"} Nov 02 03:31:13.833290 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:13.832924 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" event={"ID":"e6857c35-deec-4921-867b-e4e21ee15205","Type":"ContainerStarted","Data":"470f47fba0ad566d005a271e67bebe261c619ac3601f0cd4d650e1c2723b4038"} Nov 02 03:31:14.047770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.047721 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podStartSLOduration=2.047707054 podStartE2EDuration="2.047707054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:31:13.997893588 +0000 UTC m=+2615.993594531" watchObservedRunningTime="2026-11-02 03:31:14.047707054 +0000 UTC m=+2616.043407997" Nov 02 03:31:14.075225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.075196 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb"] Nov 02 03:31:14.178922 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.178896 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht"] Nov 02 03:31:14.656648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.654652 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:14.664687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.664658 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:14.838254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.838220 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:14.839169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:14.839147 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:31:19.076087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.076054 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" podUID="3c79e8c4-9e07-4f75-a472-367df7ec9125" containerName="payload-processing" containerID="cri-o://823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c" gracePeriod=25 Nov 02 03:31:19.179969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.179934 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" podUID="bb93c010-f500-49fb-ae98-f49c9389177a" containerName="payload-pre-processing" containerID="cri-o://681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82" gracePeriod=25 Nov 02 03:31:19.341063 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.340999 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:31:19.415429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.415391 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume\") pod \"3c79e8c4-9e07-4f75-a472-367df7ec9125\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " Nov 02 03:31:19.415604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.415485 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9\" (UniqueName: \"kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9\") pod \"3c79e8c4-9e07-4f75-a472-367df7ec9125\" (UID: \"3c79e8c4-9e07-4f75-a472-367df7ec9125\") " Nov 02 03:31:19.416810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.416780 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume" pod "3c79e8c4-9e07-4f75-a472-367df7ec9125" (UID: "3c79e8c4-9e07-4f75-a472-367df7ec9125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:31:19.418921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.418887 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9" pod "3c79e8c4-9e07-4f75-a472-367df7ec9125" (UID: "3c79e8c4-9e07-4f75-a472-367df7ec9125"). InnerVolumeSpecName "kube-api-access-xl8n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:31:19.451254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.451234 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:31:19.517028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.516999 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt\" (UniqueName: \"kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt\") pod \"bb93c010-f500-49fb-ae98-f49c9389177a\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " Nov 02 03:31:19.517220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.517188 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume\") pod \"bb93c010-f500-49fb-ae98-f49c9389177a\" (UID: \"bb93c010-f500-49fb-ae98-f49c9389177a\") " Nov 02 03:31:19.517570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.517544 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume" pod "bb93c010-f500-49fb-ae98-f49c9389177a" (UID: "bb93c010-f500-49fb-ae98-f49c9389177a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:31:19.517570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.517559 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c79e8c4-9e07-4f75-a472-367df7ec9125-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:31:19.517705 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.517580 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl8n9\" (UniqueName: \"kubernetes.io/projected/3c79e8c4-9e07-4f75-a472-367df7ec9125-kube-api-access-xl8n9\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:31:19.519143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.519119 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt" pod "bb93c010-f500-49fb-ae98-f49c9389177a" (UID: "bb93c010-f500-49fb-ae98-f49c9389177a"). InnerVolumeSpecName "kube-api-access-bs7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:31:19.619217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.619182 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7gt\" (UniqueName: \"kubernetes.io/projected/bb93c010-f500-49fb-ae98-f49c9389177a-kube-api-access-bs7gt\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:31:19.619217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.619211 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb93c010-f500-49fb-ae98-f49c9389177a-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:31:19.905941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.905855 4868 generic.go:356] "Generic (PLEG): container finished" podID="3c79e8c4-9e07-4f75-a472-367df7ec9125" containerID="823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c" exitCode=0 Nov 02 03:31:19.905941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.905904 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" Nov 02 03:31:19.906155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.905945 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" event={"ID":"3c79e8c4-9e07-4f75-a472-367df7ec9125","Type":"ContainerDied","Data":"823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c"} Nov 02 03:31:19.906155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.906029 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb" event={"ID":"3c79e8c4-9e07-4f75-a472-367df7ec9125","Type":"ContainerDied","Data":"166fb105f722ba468d193b539f1439949ab8cf86b1022a72759884b7d9644603"} Nov 02 03:31:19.906155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.906056 4868 scope.go:122] "RemoveContainer" containerID="823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c" Nov 02 03:31:19.907524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.907501 4868 generic.go:356] "Generic (PLEG): container finished" podID="bb93c010-f500-49fb-ae98-f49c9389177a" containerID="681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82" exitCode=0 Nov 02 03:31:19.907624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.907538 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" event={"ID":"bb93c010-f500-49fb-ae98-f49c9389177a","Type":"ContainerDied","Data":"681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82"} Nov 02 03:31:19.907624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.907566 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" Nov 02 03:31:19.907624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.907586 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht" event={"ID":"bb93c010-f500-49fb-ae98-f49c9389177a","Type":"ContainerDied","Data":"167f167d530d108040e0cd6e42c9129d2c79c40fecfec4b37aa641e80d76910a"} Nov 02 03:31:19.918193 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.918173 4868 scope.go:122] "RemoveContainer" containerID="823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c" Nov 02 03:31:19.920667 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:31:19.918695 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c\": container with ID starting with 823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c not found: ID does not exist" containerID="823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c" Nov 02 03:31:19.920667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.919815 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c"} err="failed to get container status \"823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c\": rpc error: code = NotFound desc = could not find container \"823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c\": container with ID starting with 823f97654a5a48409013214511426719445cb8d369448e0afd2f9627af4b947c not found: ID does not exist" Nov 02 03:31:19.920667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.919848 4868 scope.go:122] "RemoveContainer" containerID="681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82" Nov 02 03:31:19.930955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.930935 4868 scope.go:122] "RemoveContainer" containerID="681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82" Nov 02 03:31:19.931266 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:31:19.931244 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82\": container with ID starting with 681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82 not found: ID does not exist" containerID="681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82" Nov 02 03:31:19.931418 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:19.931268 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82"} err="failed to get container status \"681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82\": rpc error: code = NotFound desc = could not find container \"681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82\": container with ID starting with 681db31c7aee812a3369d23bfb0dee98314429a16ce6f5c75d155dd74c41cf82 not found: ID does not exist" Nov 02 03:31:20.070291 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:20.070252 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht"] Nov 02 03:31:20.118518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:20.118488 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a5285-599746bcc5-9g7ht"] Nov 02 03:31:20.186328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:20.186249 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb"] Nov 02 03:31:20.190592 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:20.190567 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a5285-7df9d5c5b9-zrdcb"] Nov 02 03:31:22.115489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:22.115460 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c79e8c4-9e07-4f75-a472-367df7ec9125" path="/var/lib/kubelet/pods/3c79e8c4-9e07-4f75-a472-367df7ec9125/volumes" Nov 02 03:31:22.115908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:22.115893 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb93c010-f500-49fb-ae98-f49c9389177a" path="/var/lib/kubelet/pods/bb93c010-f500-49fb-ae98-f49c9389177a/volumes" Nov 02 03:31:33.447790 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:31:33.447753 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:31:42.834131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.834093 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k"] Nov 02 03:31:42.834871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.834853 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3c79e8c4-9e07-4f75-a472-367df7ec9125" containerName="payload-processing" Nov 02 03:31:42.834916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.834872 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c79e8c4-9e07-4f75-a472-367df7ec9125" containerName="payload-processing" Nov 02 03:31:42.834916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.834908 4868 cpu_manager.go:414] "containerMap: removing container" podUID="bb93c010-f500-49fb-ae98-f49c9389177a" containerName="payload-pre-processing" Nov 02 03:31:42.834916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.834913 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb93c010-f500-49fb-ae98-f49c9389177a" containerName="payload-pre-processing" Nov 02 03:31:42.835018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.835008 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c79e8c4-9e07-4f75-a472-367df7ec9125" containerName="payload-processing" Nov 02 03:31:42.835052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.835022 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb93c010-f500-49fb-ae98-f49c9389177a" containerName="payload-pre-processing" Nov 02 03:31:42.839639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.839622 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:42.936023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.935989 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:42.936198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.936158 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:42.936286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.936267 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:42.936370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.936334 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67dnf\" (UniqueName: \"kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:42.941117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.941095 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9d5625-serving-cert\"" Nov 02 03:31:42.943871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:42.943847 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k"] Nov 02 03:31:43.037745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.037704 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.037917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.037758 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67dnf\" (UniqueName: \"kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.037917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.037819 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.037917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.037897 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.038104 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:31:43.038001 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-serving-cert: secret "maas-api-e2e-ipp-b-9d5625-serving-cert" not found Nov 02 03:31:43.038104 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:31:43.038055 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls podName:28a34277-ffca-4832-80f3-48791ef9b87f nodeName:}" failed. No retries permitted until 2026-11-02 03:31:43.538041363 +0000 UTC m=+2645.533742284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls") pod "maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" (UID: "28a34277-ffca-4832-80f3-48791ef9b87f") : secret "maas-api-e2e-ipp-b-9d5625-serving-cert" not found Nov 02 03:31:43.038890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.038867 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.040272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.040236 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.128736 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.128701 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67dnf\" (UniqueName: \"kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.549075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.549001 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.551467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.551436 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:43.752468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:43.752439 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:44.126053 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:44.126022 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k"] Nov 02 03:31:44.127049 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:31:44.127014 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a34277_ffca_4832_80f3_48791ef9b87f.slice/crio-d421de11fe7e4e1edd5fdad9a61144d62315622bde9b9a58b9dd5b06dbc9ba5c WatchSource:0}: Error finding container d421de11fe7e4e1edd5fdad9a61144d62315622bde9b9a58b9dd5b06dbc9ba5c: Status 404 returned error can't find the container with id d421de11fe7e4e1edd5fdad9a61144d62315622bde9b9a58b9dd5b06dbc9ba5c Nov 02 03:31:45.087485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:45.087447 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" event={"ID":"28a34277-ffca-4832-80f3-48791ef9b87f","Type":"ContainerStarted","Data":"d421de11fe7e4e1edd5fdad9a61144d62315622bde9b9a58b9dd5b06dbc9ba5c"} Nov 02 03:31:46.096135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:46.096094 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" event={"ID":"28a34277-ffca-4832-80f3-48791ef9b87f","Type":"ContainerStarted","Data":"697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b"} Nov 02 03:31:46.096662 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:46.096292 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:31:46.211584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:46.211535 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" podStartSLOduration=3.291394958 podStartE2EDuration="4.211520835s" totalImagesPullingTime="920.125877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:31:46.208872355 +0000 UTC m=+2648.204573298" watchObservedRunningTime="2026-11-02 03:31:46.211520835 +0000 UTC m=+2648.207221815" Nov 02 03:31:52.114585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:31:52.114497 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:32:03.467309 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:32:03.467269 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:32:09.250275 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:09.250244 4868 generic.go:356] "Generic (PLEG): container finished" podID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerID="bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa" exitCode=0 Nov 02 03:32:09.250721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:09.250314 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" event={"ID":"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4","Type":"ContainerDied","Data":"bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa"} Nov 02 03:32:09.250721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:09.250666 4868 scope.go:122] "RemoveContainer" containerID="bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa" Nov 02 03:32:09.252094 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:09.252069 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:32:09.252158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:09.252132 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:32:09.252197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:09.252173 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:32:10.264597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:10.264558 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" event={"ID":"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4","Type":"ContainerStarted","Data":"93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2"} Nov 02 03:32:10.483607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:10.483573 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:32:10.890833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:10.890801 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:32:10.892091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:10.892063 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 03:32:10.892224 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:10.892112 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 03:32:11.890979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:11.890942 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 03:32:11.891496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:11.891015 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 03:32:12.890959 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:12.890926 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 03:32:12.891132 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:12.890985 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 03:32:13.890923 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:13.890888 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 03:32:13.891084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:13.890977 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 03:32:14.890737 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:14.890701 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 03:32:14.891119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:14.890754 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 03:32:15.894695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:15.894666 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:32:15.895936 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:15.895916 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:32:33.446308 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:32:33.446268 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:32:38.429295 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:38.429263 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:32:38.434951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:38.434924 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:32:47.597884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:47.597853 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj"] Nov 02 03:32:47.598435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:47.598046 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" containerID="cri-o://3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93" gracePeriod=30 Nov 02 03:32:49.643547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:49.643509 4868 patch_prober.go:29] interesting pod/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:32:49.644019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:49.643575 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:32:52.852792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.852767 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:32:52.962237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962159 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962238 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962283 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962371 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962434 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962499 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962521 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:32:52.962763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962656 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962709 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962733 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:32:52.962887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962781 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data\") pod \"dcbb30da-81d2-4166-b377-a49e88d82e4c\" (UID: \"dcbb30da-81d2-4166-b377-a49e88d82e4c\") " Nov 02 03:32:52.962887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962868 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:32:52.962957 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.962919 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:32:52.963031 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.963006 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:32:52.963298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.963281 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:52.963400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.963304 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:52.963400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.963318 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:52.963400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.963334 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:52.963400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.963362 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbb30da-81d2-4166-b377-a49e88d82e4c-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:52.964761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.964741 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:32:52.964877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.964861 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "kube-api-access-6lcd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:32:52.965131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.965114 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:32:52.965200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:52.965128 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo" pod "dcbb30da-81d2-4166-b377-a49e88d82e4c" (UID: "dcbb30da-81d2-4166-b377-a49e88d82e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:32:53.064679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.064642 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lcd6\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-kube-api-access-6lcd6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:53.064679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.064680 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:53.064854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.064696 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:53.064854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.064712 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbb30da-81d2-4166-b377-a49e88d82e4c-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:32:53.578415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.578376 4868 generic.go:356] "Generic (PLEG): container finished" podID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerID="3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93" exitCode=0 Nov 02 03:32:53.578659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.578423 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" Nov 02 03:32:53.578659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.578452 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" event={"ID":"dcbb30da-81d2-4166-b377-a49e88d82e4c","Type":"ContainerDied","Data":"3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93"} Nov 02 03:32:53.578659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.578511 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj" event={"ID":"dcbb30da-81d2-4166-b377-a49e88d82e4c","Type":"ContainerDied","Data":"cb474e1615088e7ac6a7b8fcf9417257d9e5dca7f65759bbc99dd20c5b0da50d"} Nov 02 03:32:53.578659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.578536 4868 scope.go:122] "RemoveContainer" containerID="3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93" Nov 02 03:32:53.589168 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.589151 4868 scope.go:122] "RemoveContainer" containerID="3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93" Nov 02 03:32:53.589470 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:32:53.589452 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93\": container with ID starting with 3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93 not found: ID does not exist" containerID="3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93" Nov 02 03:32:53.589529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.589480 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93"} err="failed to get container status \"3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93\": rpc error: code = NotFound desc = could not find container \"3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93\": container with ID starting with 3d140d6e856faadfc8aea255c84b282e93cb22ed3176ac9f7059ff103d59be93 not found: ID does not exist" Nov 02 03:32:53.742074 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.742041 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj"] Nov 02 03:32:53.744988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:53.744969 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-pvrnj"] Nov 02 03:32:54.115730 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:32:54.115699 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" path="/var/lib/kubelet/pods/dcbb30da-81d2-4166-b377-a49e88d82e4c/volumes" Nov 02 03:33:03.467740 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:33:03.467701 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:33:10.991460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:10.991429 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd"] Nov 02 03:33:10.991966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:10.991952 4868 cpu_manager.go:414] "containerMap: removing container" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" Nov 02 03:33:10.992011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:10.991967 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" Nov 02 03:33:10.992099 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:10.992088 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbb30da-81d2-4166-b377-a49e88d82e4c" containerName="istio-proxy" Nov 02 03:33:10.995423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:10.995407 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.097969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.097937 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dd87f0\"/\"openshift-service-ca.crt\"" Nov 02 03:33:11.098178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.098155 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dd87f0\"/\"ipp-route-dd87f0-kserve-self-signed-certs\"" Nov 02 03:33:11.098178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.097963 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dd87f0\"/\"kube-root-ca.crt\"" Nov 02 03:33:11.098178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.098078 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dd87f0\"/\"default-dockercfg-f6psz\"" Nov 02 03:33:11.100734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.100711 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd"] Nov 02 03:33:11.119511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.119483 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.119609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.119549 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.119657 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.119626 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.119735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.119719 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdn8b\" (UniqueName: \"kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.119784 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.119758 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.119825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.119809 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.221635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.221591 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdn8b\" (UniqueName: \"kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.221817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.221669 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.221817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.221742 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.221908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.221819 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.221908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.221881 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.221981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.221915 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.222197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.222168 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.222325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.222245 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.222412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.222329 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.224484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.224456 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.224990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.224970 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.278516 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.278445 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdn8b\" (UniqueName: \"kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b\") pod \"ipp-route-dd87f0-kserve-54d77955c-znbrd\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.308499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.308451 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:11.775878 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.775857 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd"] Nov 02 03:33:11.777907 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:33:11.777878 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadd71940_ba82_4b77_8e3d_ff5efe0a5b71.slice/crio-5659c0731e70346ca00ca8fcfa50c309a631fc812b11bcf45aab2e5c3d6b26cc WatchSource:0}: Error finding container 5659c0731e70346ca00ca8fcfa50c309a631fc812b11bcf45aab2e5c3d6b26cc: Status 404 returned error can't find the container with id 5659c0731e70346ca00ca8fcfa50c309a631fc812b11bcf45aab2e5c3d6b26cc Nov 02 03:33:11.933680 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.933641 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth"] Nov 02 03:33:11.933880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:11.933838 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" podUID="9f199224-aa72-4c40-8d80-b5b587b8c8b3" containerName="maas-api" containerID="cri-o://0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904" gracePeriod=30 Nov 02 03:33:12.304177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.304117 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:33:12.338918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.337748 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\") pod \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " Nov 02 03:33:12.338918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.337872 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs\") pod \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " Nov 02 03:33:12.338918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.337933 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca\") pod \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " Nov 02 03:33:12.338918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.337983 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln\" (UniqueName: \"kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln\") pod \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\" (UID: \"9f199224-aa72-4c40-8d80-b5b587b8c8b3\") " Nov 02 03:33:12.351577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.351536 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca" pod "9f199224-aa72-4c40-8d80-b5b587b8c8b3" (UID: "9f199224-aa72-4c40-8d80-b5b587b8c8b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:33:12.376030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.375966 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln" pod "9f199224-aa72-4c40-8d80-b5b587b8c8b3" (UID: "9f199224-aa72-4c40-8d80-b5b587b8c8b3"). InnerVolumeSpecName "kube-api-access-v46ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:33:12.376233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.376212 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls" pod "9f199224-aa72-4c40-8d80-b5b587b8c8b3" (UID: "9f199224-aa72-4c40-8d80-b5b587b8c8b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:33:12.381380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.376568 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs" pod "9f199224-aa72-4c40-8d80-b5b587b8c8b3" (UID: "9f199224-aa72-4c40-8d80-b5b587b8c8b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:33:12.440469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.440424 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46ln\" (UniqueName: \"kubernetes.io/projected/9f199224-aa72-4c40-8d80-b5b587b8c8b3-kube-api-access-v46ln\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:12.440469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.440470 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:12.440684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.440485 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f199224-aa72-4c40-8d80-b5b587b8c8b3-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:12.440684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.440499 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f199224-aa72-4c40-8d80-b5b587b8c8b3-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:12.733944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.733912 4868 generic.go:356] "Generic (PLEG): container finished" podID="e6857c35-deec-4921-867b-e4e21ee15205" containerID="2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f" exitCode=0 Nov 02 03:33:12.734130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.733988 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" event={"ID":"e6857c35-deec-4921-867b-e4e21ee15205","Type":"ContainerDied","Data":"2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f"} Nov 02 03:33:12.734472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.734449 4868 scope.go:122] "RemoveContainer" containerID="2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f" Nov 02 03:33:12.735678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.735650 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" event={"ID":"add71940-ba82-4b77-8e3d-ff5efe0a5b71","Type":"ContainerStarted","Data":"0fba4fe31423fa1223761902e62de9ad312c771c72519865839f15879286dc99"} Nov 02 03:33:12.735795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.735700 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" event={"ID":"add71940-ba82-4b77-8e3d-ff5efe0a5b71","Type":"ContainerStarted","Data":"5659c0731e70346ca00ca8fcfa50c309a631fc812b11bcf45aab2e5c3d6b26cc"} Nov 02 03:33:12.736115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.736084 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:33:12.736197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.736155 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:33:12.736197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.736187 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:33:12.737111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.737087 4868 generic.go:356] "Generic (PLEG): container finished" podID="9f199224-aa72-4c40-8d80-b5b587b8c8b3" containerID="0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904" exitCode=0 Nov 02 03:33:12.737196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.737114 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" event={"ID":"9f199224-aa72-4c40-8d80-b5b587b8c8b3","Type":"ContainerDied","Data":"0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904"} Nov 02 03:33:12.737196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.737128 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" Nov 02 03:33:12.737196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.737161 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth" event={"ID":"9f199224-aa72-4c40-8d80-b5b587b8c8b3","Type":"ContainerDied","Data":"4dbe162fbf0f87b552d92d6eb7ab195cbb354cd049f16ff408bc0e9e75ff4dc1"} Nov 02 03:33:12.737196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.737184 4868 scope.go:122] "RemoveContainer" containerID="0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904" Nov 02 03:33:12.748111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.748090 4868 scope.go:122] "RemoveContainer" containerID="0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904" Nov 02 03:33:12.748391 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:33:12.748368 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904\": container with ID starting with 0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904 not found: ID does not exist" containerID="0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904" Nov 02 03:33:12.748452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.748399 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904"} err="failed to get container status \"0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904\": rpc error: code = NotFound desc = could not find container \"0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904\": container with ID starting with 0cf1ea3ed579be47333a98615a7b6e33e979ea069043b5d8ec5a54c249128904 not found: ID does not exist" Nov 02 03:33:12.964557 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.964505 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth"] Nov 02 03:33:12.971594 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:12.969166 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a5285-d8954f6c-9ngth"] Nov 02 03:33:13.081061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:13.080944 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" podStartSLOduration=3.080912859 podStartE2EDuration="3.080912859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:33:13.079158058 +0000 UTC m=+2735.074859000" watchObservedRunningTime="2026-11-02 03:33:13.080912859 +0000 UTC m=+2735.076613804" Nov 02 03:33:13.308648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:13.308615 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:13.763007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:13.762967 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" event={"ID":"e6857c35-deec-4921-867b-e4e21ee15205","Type":"ContainerStarted","Data":"54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8"} Nov 02 03:33:14.120669 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:14.120536 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f199224-aa72-4c40-8d80-b5b587b8c8b3" path="/var/lib/kubelet/pods/9f199224-aa72-4c40-8d80-b5b587b8c8b3/volumes" Nov 02 03:33:14.659082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:14.654650 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:33:14.659624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:14.659333 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 03:33:14.659624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:14.659407 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 03:33:14.851480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:14.851446 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:33:15.656771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:15.656733 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 03:33:15.657064 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:15.657023 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 03:33:16.657792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:16.657756 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 03:33:16.658395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:16.658365 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 03:33:17.656382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:17.655668 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 03:33:17.656382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:17.655744 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 03:33:18.654374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:18.653647 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 03:33:18.654374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:18.653703 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 03:33:19.657469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:19.657439 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:33:19.658535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:19.658513 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:33:20.878319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:20.878284 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb"] Nov 02 03:33:20.878753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:20.878503 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" containerID="cri-o://89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13" gracePeriod=30 Nov 02 03:33:21.308894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:21.308817 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:21.321423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:21.321398 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:21.822808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:21.822776 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:33:26.143902 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.143849 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:33:26.213963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.213926 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214092 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214243 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214177 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214243 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214227 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214312 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214401 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214434 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214469 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.214635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.214513 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token\") pod \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\" (UID: \"27cfc3b9-7a65-4bb0-8245-ae992cde49d0\") " Nov 02 03:33:26.215304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.215275 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:26.215562 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.215507 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:26.215692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.215591 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:26.215870 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.215709 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:26.217444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.216449 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:33:26.221314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.221290 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:33:26.221314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.221297 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:26.221516 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.221401 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:33:26.221692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.221668 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj" pod "27cfc3b9-7a65-4bb0-8245-ae992cde49d0" (UID: "27cfc3b9-7a65-4bb0-8245-ae992cde49d0"). InnerVolumeSpecName "kube-api-access-ws5sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:33:26.316013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.315985 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316011 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316025 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws5sj\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-kube-api-access-ws5sj\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316038 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316051 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316063 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316075 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316086 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.316186 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.316099 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27cfc3b9-7a65-4bb0-8245-ae992cde49d0-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:26.842929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.842896 4868 generic.go:356] "Generic (PLEG): container finished" podID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerID="89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13" exitCode=0 Nov 02 03:33:26.842929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.842932 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" event={"ID":"27cfc3b9-7a65-4bb0-8245-ae992cde49d0","Type":"ContainerDied","Data":"89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13"} Nov 02 03:33:26.843177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.842947 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" Nov 02 03:33:26.843177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.842973 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb" event={"ID":"27cfc3b9-7a65-4bb0-8245-ae992cde49d0","Type":"ContainerDied","Data":"07aef3ab26ab7fff295017a10a73196bd68223ef6ca4187f030874f0947161ad"} Nov 02 03:33:26.843177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.842997 4868 scope.go:122] "RemoveContainer" containerID="89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13" Nov 02 03:33:26.853334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.853309 4868 scope.go:122] "RemoveContainer" containerID="17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e" Nov 02 03:33:26.865831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.865809 4868 scope.go:122] "RemoveContainer" containerID="89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13" Nov 02 03:33:26.866089 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:33:26.866067 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13\": container with ID starting with 89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13 not found: ID does not exist" containerID="89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13" Nov 02 03:33:26.866164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.866103 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13"} err="failed to get container status \"89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13\": rpc error: code = NotFound desc = could not find container \"89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13\": container with ID starting with 89863be8e9ed8f7b8eb95b2737bbd025b33140bdb9afafb6de4cba4631501c13 not found: ID does not exist" Nov 02 03:33:26.866164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.866129 4868 scope.go:122] "RemoveContainer" containerID="17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e" Nov 02 03:33:26.866479 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:33:26.866460 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e\": container with ID starting with 17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e not found: ID does not exist" containerID="17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e" Nov 02 03:33:26.866528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:26.866486 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e"} err="failed to get container status \"17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e\": rpc error: code = NotFound desc = could not find container \"17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e\": container with ID starting with 17af252d71a802d259ece4228dcc0a58af8f7850969e31d9a030e22a699a7f1e not found: ID does not exist" Nov 02 03:33:27.005333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:27.005304 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb"] Nov 02 03:33:27.056543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:27.056512 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6a5285-openshift-default-6596676675-lfxqb"] Nov 02 03:33:28.123052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:28.123019 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" path="/var/lib/kubelet/pods/27cfc3b9-7a65-4bb0-8245-ae992cde49d0/volumes" Nov 02 03:33:33.456555 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:33:33.456509 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:33:39.177220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177187 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th"] Nov 02 03:33:39.177685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177671 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9f199224-aa72-4c40-8d80-b5b587b8c8b3" containerName="maas-api" Nov 02 03:33:39.177727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177686 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f199224-aa72-4c40-8d80-b5b587b8c8b3" containerName="maas-api" Nov 02 03:33:39.177727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177721 4868 cpu_manager.go:414] "containerMap: removing container" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" Nov 02 03:33:39.177805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177727 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" Nov 02 03:33:39.177805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177746 4868 cpu_manager.go:414] "containerMap: removing container" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" Nov 02 03:33:39.177805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177753 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" Nov 02 03:33:39.177897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177827 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f199224-aa72-4c40-8d80-b5b587b8c8b3" containerName="maas-api" Nov 02 03:33:39.177897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.177835 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" Nov 02 03:33:39.181969 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.181948 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.283502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.283471 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d94cc3\"/\"openshift-service-ca.crt\"" Nov 02 03:33:39.283659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.283557 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d94cc3\"/\"kube-root-ca.crt\"" Nov 02 03:33:39.283806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.283793 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d94cc3\"/\"tenantcfg-model-d94cc3-kserve-self-signed-certs\"" Nov 02 03:33:39.283868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.283852 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d94cc3\"/\"default-dockercfg-v7gk8\"" Nov 02 03:33:39.286296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.286278 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th"] Nov 02 03:33:39.374601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.371297 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.374601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.371427 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njkwm\" (UniqueName: \"kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.374601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.371527 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.374601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.371564 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.374601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.371617 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.374601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.371677 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.472589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472496 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.472589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472532 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.472589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472560 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.472865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472595 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.472865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472670 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.472865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472742 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njkwm\" (UniqueName: \"kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.473029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472962 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.473029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.472979 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.473135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.473088 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.475396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.475338 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.475610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.475367 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.529145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.529112 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njkwm\" (UniqueName: \"kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm\") pod \"tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:39.796424 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:39.795916 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:40.178571 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:40.178545 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th"] Nov 02 03:33:40.179330 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:33:40.179306 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1e6c082_411e_42c1_972f_17d3b7c0a1b0.slice/crio-a264964cb1e49a9a530bbc264e6619d1d1574d6100cd4862100bd70cf91cd0f2 WatchSource:0}: Error finding container a264964cb1e49a9a530bbc264e6619d1d1574d6100cd4862100bd70cf91cd0f2: Status 404 returned error can't find the container with id a264964cb1e49a9a530bbc264e6619d1d1574d6100cd4862100bd70cf91cd0f2 Nov 02 03:33:40.955169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:40.955132 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" event={"ID":"e1e6c082-411e-42c1-972f-17d3b7c0a1b0","Type":"ContainerStarted","Data":"af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636"} Nov 02 03:33:40.955169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:40.955179 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" event={"ID":"e1e6c082-411e-42c1-972f-17d3b7c0a1b0","Type":"ContainerStarted","Data":"a264964cb1e49a9a530bbc264e6619d1d1574d6100cd4862100bd70cf91cd0f2"} Nov 02 03:33:41.070696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:41.070643 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" podStartSLOduration=2.070629489 podStartE2EDuration="2.070629489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:33:41.069431105 +0000 UTC m=+2763.065132049" watchObservedRunningTime="2026-11-02 03:33:41.070629489 +0000 UTC m=+2763.066330432" Nov 02 03:33:41.796639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:41.796604 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:44.703601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:44.703557 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7"] Nov 02 03:33:44.704073 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:44.703815 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" containerID="cri-o://9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b" gracePeriod=30 Nov 02 03:33:49.797341 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:49.797305 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:49.809668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:49.809643 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:49.966106 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:49.966082 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:33:50.011131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.011056 4868 generic.go:356] "Generic (PLEG): container finished" podID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerID="9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b" exitCode=0 Nov 02 03:33:50.011131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.011104 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" Nov 02 03:33:50.011286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.011145 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" event={"ID":"14b1a546-10f6-4e88-bbae-492093fdc38d","Type":"ContainerDied","Data":"9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b"} Nov 02 03:33:50.011286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.011199 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7" event={"ID":"14b1a546-10f6-4e88-bbae-492093fdc38d","Type":"ContainerDied","Data":"8e9a5a6fab0236cd7f472c0604062671cc4bff12c0d53eebf3635e82daf3a35c"} Nov 02 03:33:50.011286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.011224 4868 scope.go:122] "RemoveContainer" containerID="9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b" Nov 02 03:33:50.022596 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.022578 4868 scope.go:122] "RemoveContainer" containerID="9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b" Nov 02 03:33:50.022873 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:33:50.022852 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b\": container with ID starting with 9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b not found: ID does not exist" containerID="9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b" Nov 02 03:33:50.022923 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.022884 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b"} err="failed to get container status \"9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b\": rpc error: code = NotFound desc = could not find container \"9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b\": container with ID starting with 9ae6bc231671345cda9ea6a3abd185817294cb927359a03994967c82d107777b not found: ID does not exist" Nov 02 03:33:50.025089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.025065 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:33:50.035008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.034988 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035020 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035098 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035168 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035189 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035221 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035229 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:50.035449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035307 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:50.035449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035390 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035424 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035429 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:50.035660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035465 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy\") pod \"14b1a546-10f6-4e88-bbae-492093fdc38d\" (UID: \"14b1a546-10f6-4e88-bbae-492093fdc38d\") " Nov 02 03:33:50.035660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035568 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:50.035660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035585 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:33:50.035987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035969 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.036077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.035991 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b1a546-10f6-4e88-bbae-492093fdc38d-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.036077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.036005 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.036077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.036019 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.036077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.036031 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.037338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.037322 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "kube-api-access-77smc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:33:50.037517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.037502 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:33:50.037892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.037871 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:33:50.037947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.037901 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token" pod "14b1a546-10f6-4e88-bbae-492093fdc38d" (UID: "14b1a546-10f6-4e88-bbae-492093fdc38d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:33:50.137096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.137056 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.137096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.137083 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.137096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.137098 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b1a546-10f6-4e88-bbae-492093fdc38d-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.137388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.137112 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77smc\" (UniqueName: \"kubernetes.io/projected/14b1a546-10f6-4e88-bbae-492093fdc38d-kube-api-access-77smc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:33:50.427937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.427900 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7"] Nov 02 03:33:50.431817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:50.431784 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7sb7"] Nov 02 03:33:52.115163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:33:52.115129 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" path="/var/lib/kubelet/pods/14b1a546-10f6-4e88-bbae-492093fdc38d/volumes" Nov 02 03:34:03.446300 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:34:03.446258 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:34:08.105002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.104963 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877"] Nov 02 03:34:08.105421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.105180 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" podUID="7dc4386b-1062-4e33-aff2-a14067e6831e" containerName="maas-api" containerID="cri-o://eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de" gracePeriod=30 Nov 02 03:34:08.435475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.435448 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:34:08.444846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.444818 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs\") pod \"7dc4386b-1062-4e33-aff2-a14067e6831e\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " Nov 02 03:34:08.444949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.444880 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca\") pod \"7dc4386b-1062-4e33-aff2-a14067e6831e\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " Nov 02 03:34:08.445111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.445090 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\") pod \"7dc4386b-1062-4e33-aff2-a14067e6831e\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " Nov 02 03:34:08.445196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.445170 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp\" (UniqueName: \"kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp\") pod \"7dc4386b-1062-4e33-aff2-a14067e6831e\" (UID: \"7dc4386b-1062-4e33-aff2-a14067e6831e\") " Nov 02 03:34:08.445286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.445259 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca" pod "7dc4386b-1062-4e33-aff2-a14067e6831e" (UID: "7dc4386b-1062-4e33-aff2-a14067e6831e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:34:08.445590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.445565 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc4386b-1062-4e33-aff2-a14067e6831e-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:34:08.447020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.446989 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs" pod "7dc4386b-1062-4e33-aff2-a14067e6831e" (UID: "7dc4386b-1062-4e33-aff2-a14067e6831e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:34:08.447663 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.447638 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls" pod "7dc4386b-1062-4e33-aff2-a14067e6831e" (UID: "7dc4386b-1062-4e33-aff2-a14067e6831e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:34:08.447754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.447698 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp" pod "7dc4386b-1062-4e33-aff2-a14067e6831e" (UID: "7dc4386b-1062-4e33-aff2-a14067e6831e"). InnerVolumeSpecName "kube-api-access-xs7zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:34:08.554375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.550441 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:34:08.554375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.550482 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7zp\" (UniqueName: \"kubernetes.io/projected/7dc4386b-1062-4e33-aff2-a14067e6831e-kube-api-access-xs7zp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:34:08.554375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:08.550500 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dc4386b-1062-4e33-aff2-a14067e6831e-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:34:09.224454 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.224417 4868 generic.go:356] "Generic (PLEG): container finished" podID="7dc4386b-1062-4e33-aff2-a14067e6831e" containerID="eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de" exitCode=0 Nov 02 03:34:09.224981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.224508 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" event={"ID":"7dc4386b-1062-4e33-aff2-a14067e6831e","Type":"ContainerDied","Data":"eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de"} Nov 02 03:34:09.224981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.224588 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" event={"ID":"7dc4386b-1062-4e33-aff2-a14067e6831e","Type":"ContainerDied","Data":"302a6a18c872747240c53ec609010dc461dd5280c799fb62a10817226b4d3e93"} Nov 02 03:34:09.224981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.224618 4868 scope.go:122] "RemoveContainer" containerID="eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de" Nov 02 03:34:09.224981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.224780 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877" Nov 02 03:34:09.244131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.244108 4868 scope.go:122] "RemoveContainer" containerID="eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de" Nov 02 03:34:09.245559 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:34:09.245529 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de\": container with ID starting with eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de not found: ID does not exist" containerID="eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de" Nov 02 03:34:09.245683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.245567 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de"} err="failed to get container status \"eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de\": rpc error: code = NotFound desc = could not find container \"eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de\": container with ID starting with eda754515563ba808debfdc757ba253ed9c82c98e33dbfa7ad65da2e23bb61de not found: ID does not exist" Nov 02 03:34:09.387933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.387898 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877"] Nov 02 03:34:09.390092 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:09.390070 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-045653-88cd885cb-ww877"] Nov 02 03:34:10.117068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:34:10.117033 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc4386b-1062-4e33-aff2-a14067e6831e" path="/var/lib/kubelet/pods/7dc4386b-1062-4e33-aff2-a14067e6831e/volumes" Nov 02 03:34:33.445466 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:34:33.445428 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:35:03.446302 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:35:03.446268 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:35:32.307187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.307154 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k"] Nov 02 03:35:32.307703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.307367 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" podUID="28a34277-ffca-4832-80f3-48791ef9b87f" containerName="maas-api" containerID="cri-o://697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b" gracePeriod=30 Nov 02 03:35:32.660461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.660435 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:35:32.748670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.748631 4868 generic.go:356] "Generic (PLEG): container finished" podID="28a34277-ffca-4832-80f3-48791ef9b87f" containerID="697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b" exitCode=0 Nov 02 03:35:32.749039 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.749009 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" event={"ID":"28a34277-ffca-4832-80f3-48791ef9b87f","Type":"ContainerDied","Data":"697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b"} Nov 02 03:35:32.749265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.749246 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" event={"ID":"28a34277-ffca-4832-80f3-48791ef9b87f","Type":"ContainerDied","Data":"d421de11fe7e4e1edd5fdad9a61144d62315622bde9b9a58b9dd5b06dbc9ba5c"} Nov 02 03:35:32.749413 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.749398 4868 scope.go:122] "RemoveContainer" containerID="697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b" Nov 02 03:35:32.749689 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.749671 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k" Nov 02 03:35:32.762915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.762869 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca\") pod \"28a34277-ffca-4832-80f3-48791ef9b87f\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " Nov 02 03:35:32.763169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.763148 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf\" (UniqueName: \"kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf\") pod \"28a34277-ffca-4832-80f3-48791ef9b87f\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " Nov 02 03:35:32.763434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.763411 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs\") pod \"28a34277-ffca-4832-80f3-48791ef9b87f\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " Nov 02 03:35:32.763646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.763628 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\") pod \"28a34277-ffca-4832-80f3-48791ef9b87f\" (UID: \"28a34277-ffca-4832-80f3-48791ef9b87f\") " Nov 02 03:35:32.766627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.766304 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca" pod "28a34277-ffca-4832-80f3-48791ef9b87f" (UID: "28a34277-ffca-4832-80f3-48791ef9b87f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:35:32.770851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.770809 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf" pod "28a34277-ffca-4832-80f3-48791ef9b87f" (UID: "28a34277-ffca-4832-80f3-48791ef9b87f"). InnerVolumeSpecName "kube-api-access-67dnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:35:32.784394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.783051 4868 scope.go:122] "RemoveContainer" containerID="697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b" Nov 02 03:35:32.784394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.783092 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs" pod "28a34277-ffca-4832-80f3-48791ef9b87f" (UID: "28a34277-ffca-4832-80f3-48791ef9b87f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:35:32.784617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.784408 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls" pod "28a34277-ffca-4832-80f3-48791ef9b87f" (UID: "28a34277-ffca-4832-80f3-48791ef9b87f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:35:32.784617 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:35:32.784474 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b\": container with ID starting with 697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b not found: ID does not exist" containerID="697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b" Nov 02 03:35:32.784617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.784527 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b"} err="failed to get container status \"697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b\": rpc error: code = NotFound desc = could not find container \"697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b\": container with ID starting with 697535c5a939690867a571b058b8a1dc0ca40e42b334081149d6c0b62bed773b not found: ID does not exist" Nov 02 03:35:32.865703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.865675 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:32.865703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.865704 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28a34277-ffca-4832-80f3-48791ef9b87f-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:32.865703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.865716 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a34277-ffca-4832-80f3-48791ef9b87f-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:32.865927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:32.865732 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67dnf\" (UniqueName: \"kubernetes.io/projected/28a34277-ffca-4832-80f3-48791ef9b87f-kube-api-access-67dnf\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:33.170117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:33.170092 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k"] Nov 02 03:35:33.175464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:33.175437 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9d5625-7ccfcd44d7-5422k"] Nov 02 03:35:33.446898 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:35:33.446862 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:35:34.115465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:34.115435 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a34277-ffca-4832-80f3-48791ef9b87f" path="/var/lib/kubelet/pods/28a34277-ffca-4832-80f3-48791ef9b87f/volumes" Nov 02 03:35:40.883942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:40.883911 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78"] Nov 02 03:35:40.884417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:40.884106 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" containerID="cri-o://93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2" gracePeriod=30 Nov 02 03:35:45.898665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:45.898630 4868 patch_prober.go:29] interesting pod/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": read tcp 10.132.0.2:54080->10.132.0.105:15021: read: connection reset by peer" start-of-body= Nov 02 03:35:45.899083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:45.898692 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": read tcp 10.132.0.2:54080->10.132.0.105:15021: read: connection reset by peer" Nov 02 03:35:46.151572 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.151521 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:35:46.232700 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232674 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.232846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232748 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.232846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232780 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.232967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232873 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.232967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232906 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.232967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232946 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:35:46.233135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.232948 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.233135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233007 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:35:46.233135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233016 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.233135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233108 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.233333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233141 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:35:46.233489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233469 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:35:46.233602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233579 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:35:46.233652 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233601 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.233652 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233621 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.233652 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233636 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.233773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.233650 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.235230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.235206 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "kube-api-access-ljnk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:35:46.235333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.235241 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:35:46.235648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.235633 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:35:46.334523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.334500 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy\") pod \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\" (UID: \"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4\") " Nov 02 03:35:46.334948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.334930 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.335025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.334950 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.335025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.334960 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.335025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.334969 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnk5\" (UniqueName: \"kubernetes.io/projected/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-kube-api-access-ljnk5\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.336490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.336469 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy" pod "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" (UID: "7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:35:46.435699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.435624 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:46.859325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.859293 4868 generic.go:356] "Generic (PLEG): container finished" podID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerID="93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2" exitCode=0 Nov 02 03:35:46.859503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.859346 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" Nov 02 03:35:46.859503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.859398 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" event={"ID":"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4","Type":"ContainerDied","Data":"93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2"} Nov 02 03:35:46.859503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.859456 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78" event={"ID":"7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4","Type":"ContainerDied","Data":"1c3e346c02d51ee3840d11424376b05305b0c695fb4627940d45587f9ea7b9d4"} Nov 02 03:35:46.859503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.859486 4868 scope.go:122] "RemoveContainer" containerID="93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2" Nov 02 03:35:46.870245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.870199 4868 scope.go:122] "RemoveContainer" containerID="bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa" Nov 02 03:35:46.882316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.882296 4868 scope.go:122] "RemoveContainer" containerID="93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2" Nov 02 03:35:46.882627 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:35:46.882606 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2\": container with ID starting with 93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2 not found: ID does not exist" containerID="93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2" Nov 02 03:35:46.882931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.882638 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2"} err="failed to get container status \"93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2\": rpc error: code = NotFound desc = could not find container \"93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2\": container with ID starting with 93ba4ac3b5f0f204a60bd7f68a5cf2189df13dfce9225de1fc0237f3455c5be2 not found: ID does not exist" Nov 02 03:35:46.882931 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.882655 4868 scope.go:122] "RemoveContainer" containerID="bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa" Nov 02 03:35:46.882931 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:35:46.882913 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa\": container with ID starting with bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa not found: ID does not exist" containerID="bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa" Nov 02 03:35:46.883126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:46.882935 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa"} err="failed to get container status \"bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa\": rpc error: code = NotFound desc = could not find container \"bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa\": container with ID starting with bc81ddd03346c3a660f77c7b0c8b317e81baca13ded3bc0473f86c83b82e38aa not found: ID does not exist" Nov 02 03:35:47.105440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:47.105407 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78"] Nov 02 03:35:47.154679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:47.154591 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d5625-openshift-default-667b78d6c9-m4m78"] Nov 02 03:35:48.114297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:48.114260 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" path="/var/lib/kubelet/pods/7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4/volumes" Nov 02 03:35:49.034238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.034207 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb"] Nov 02 03:35:49.034525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.034470 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" podUID="c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" containerName="maas-api" containerID="cri-o://28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe" gracePeriod=30 Nov 02 03:35:49.354919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.354892 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:35:49.471320 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.471286 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca\") pod \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " Nov 02 03:35:49.471655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.471634 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs\") pod \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " Nov 02 03:35:49.474169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.474100 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg\" (UniqueName: \"kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg\") pod \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " Nov 02 03:35:49.474382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.474345 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\") pod \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\" (UID: \"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71\") " Nov 02 03:35:49.475549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.475174 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca" pod "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" (UID: "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:35:49.485019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.484958 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs" pod "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" (UID: "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:35:49.515668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.515633 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls" pod "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" (UID: "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:35:49.516303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.516256 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg" pod "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" (UID: "c4bf38d8-4c58-4b5d-87e5-0c1432af7f71"). InnerVolumeSpecName "kube-api-access-zl2rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:35:49.576499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.576467 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:49.576499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.576496 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2rg\" (UniqueName: \"kubernetes.io/projected/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-kube-api-access-zl2rg\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:49.576685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.576508 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:49.576685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.576517 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:35:49.877907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.877872 4868 generic.go:356] "Generic (PLEG): container finished" podID="c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" containerID="28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe" exitCode=0 Nov 02 03:35:49.878084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.877916 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" Nov 02 03:35:49.878084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.877949 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" event={"ID":"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71","Type":"ContainerDied","Data":"28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe"} Nov 02 03:35:49.878084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.878005 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb" event={"ID":"c4bf38d8-4c58-4b5d-87e5-0c1432af7f71","Type":"ContainerDied","Data":"91fa3af300bcb15a78a6c95fa2d00219c78e4e09db3776ce2e8c231c4b5f20a0"} Nov 02 03:35:49.878084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.878060 4868 scope.go:122] "RemoveContainer" containerID="28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe" Nov 02 03:35:49.888402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.888383 4868 scope.go:122] "RemoveContainer" containerID="28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe" Nov 02 03:35:49.888651 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:35:49.888624 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe\": container with ID starting with 28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe not found: ID does not exist" containerID="28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe" Nov 02 03:35:49.888712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:49.888659 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe"} err="failed to get container status \"28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe\": rpc error: code = NotFound desc = could not find container \"28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe\": container with ID starting with 28e5f7ea74dd208617dd997aa2f48dd45d1f3c70bf5aada09562b7e51961ddbe not found: ID does not exist" Nov 02 03:35:50.043544 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.043513 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb"] Nov 02 03:35:50.048148 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.048121 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f1a692-7b4599d644-wzjcb"] Nov 02 03:35:50.185770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.185110 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" path="/var/lib/kubelet/pods/c4bf38d8-4c58-4b5d-87e5-0c1432af7f71/volumes" Nov 02 03:35:50.550527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.550435 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4"] Nov 02 03:35:50.551703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.551679 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" containerName="maas-api" Nov 02 03:35:50.551849 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.551836 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" containerName="maas-api" Nov 02 03:35:50.551953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.551943 4868 cpu_manager.go:414] "containerMap: removing container" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552033 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552077 4868 cpu_manager.go:414] "containerMap: removing container" podUID="28a34277-ffca-4832-80f3-48791ef9b87f" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552083 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a34277-ffca-4832-80f3-48791ef9b87f" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552116 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552123 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552133 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552140 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552152 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7dc4386b-1062-4e33-aff2-a14067e6831e" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552159 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc4386b-1062-4e33-aff2-a14067e6831e" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552283 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bf38d8-4c58-4b5d-87e5-0c1432af7f71" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552294 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b1a546-10f6-4e88-bbae-492093fdc38d" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552304 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="27cfc3b9-7a65-4bb0-8245-ae992cde49d0" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552312 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc4386b-1062-4e33-aff2-a14067e6831e" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552322 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552333 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a34277-ffca-4832-80f3-48791ef9b87f" containerName="maas-api" Nov 02 03:35:50.552363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.552340 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5cac0b-7b7c-4b11-aafb-bf00572a2ba4" containerName="istio-proxy" Nov 02 03:35:50.560180 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.560157 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.595803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.595773 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4"] Nov 02 03:35:50.598947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.598922 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh5x\" (UniqueName: \"kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x\") pod \"payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.599057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.598963 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.612778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.612752 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e44998\"" Nov 02 03:35:50.612893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.612803 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e44998-dockercfg-7r7m9\"" Nov 02 03:35:50.700575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.700542 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh5x\" (UniqueName: \"kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x\") pod \"payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.700704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.700591 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.701301 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.701280 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.756761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.756730 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh5x\" (UniqueName: \"kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x\") pod \"payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:50.875215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:50.875191 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:51.252949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:51.252834 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4"] Nov 02 03:35:51.256227 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:35:51.256197 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486e894c_3ba6_4c5d_9c65_09338df48932.slice/crio-e1a3753bc8e591edef1d579576ea62b5de8bdd827065efb7d33ba65c986d273b WatchSource:0}: Error finding container e1a3753bc8e591edef1d579576ea62b5de8bdd827065efb7d33ba65c986d273b: Status 404 returned error can't find the container with id e1a3753bc8e591edef1d579576ea62b5de8bdd827065efb7d33ba65c986d273b Nov 02 03:35:51.891066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:51.891037 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" event={"ID":"486e894c-3ba6-4c5d-9c65-09338df48932","Type":"ContainerStarted","Data":"2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654"} Nov 02 03:35:51.891478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:51.891085 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" event={"ID":"486e894c-3ba6-4c5d-9c65-09338df48932","Type":"ContainerStarted","Data":"e1a3753bc8e591edef1d579576ea62b5de8bdd827065efb7d33ba65c986d273b"} Nov 02 03:35:51.891478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:51.891166 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:35:52.003381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:52.003319 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" podStartSLOduration=2.003304688 podStartE2EDuration="2.003304688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:35:52.002543746 +0000 UTC m=+2893.998244689" watchObservedRunningTime="2026-11-02 03:35:52.003304688 +0000 UTC m=+2893.999005630" Nov 02 03:35:58.168747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:58.168031 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9"] Nov 02 03:35:58.168747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:35:58.168233 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" containerID="cri-o://f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68" gracePeriod=30 Nov 02 03:36:00.119303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.119262 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt"] Nov 02 03:36:00.124011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.123989 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.279383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.278535 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0a04e0-openshift-default-dockercfg-ntllw\"" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.301418 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt"] Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302002 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302056 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qm8g\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302135 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302166 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302217 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302688 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302830 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302908 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.305384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.302938 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.403986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.403889 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.403986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.403946 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.403991 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404104 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404170 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404218 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404244 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404318 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404342 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qm8g\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404381 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404485 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404564 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.404675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.404667 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.405288 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.405263 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.406699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.406669 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.407068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.407043 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.461130 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.461098 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qm8g\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.461304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.461139 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token\") pod \"e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:00.739343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:00.739251 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:01.222638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:01.222607 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt"] Nov 02 03:36:01.225011 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:36:01.224971 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35fd55b_dbd5_4fa0_9782_fd36825c9850.slice/crio-7eee050d2ffd99391d27b13f88c888f6b1a0cda023d56abbba0c7a30b69fb6e0 WatchSource:0}: Error finding container 7eee050d2ffd99391d27b13f88c888f6b1a0cda023d56abbba0c7a30b69fb6e0: Status 404 returned error can't find the container with id 7eee050d2ffd99391d27b13f88c888f6b1a0cda023d56abbba0c7a30b69fb6e0 Nov 02 03:36:01.226854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:01.226818 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:36:01.226946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:01.226886 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:36:01.226946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:01.226917 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:36:01.956160 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:01.956118 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" event={"ID":"e35fd55b-dbd5-4fa0-9782-fd36825c9850","Type":"ContainerStarted","Data":"4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461"} Nov 02 03:36:01.956344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:01.956178 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" event={"ID":"e35fd55b-dbd5-4fa0-9782-fd36825c9850","Type":"ContainerStarted","Data":"7eee050d2ffd99391d27b13f88c888f6b1a0cda023d56abbba0c7a30b69fb6e0"} Nov 02 03:36:02.123940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:02.123889 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podStartSLOduration=2.123875459 podStartE2EDuration="2.123875459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:36:02.122409048 +0000 UTC m=+2904.118109990" watchObservedRunningTime="2026-11-02 03:36:02.123875459 +0000 UTC m=+2904.119576462" Nov 02 03:36:02.739965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:02.739934 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:02.745600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:02.745577 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:02.898904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:02.898879 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:36:02.961408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:02.961383 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:02.962278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:02.962257 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:36:03.421419 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.421389 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:36:03.444785 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:03.444723 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:36:03.537542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537513 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537554 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537630 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537669 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537691 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537719 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537799 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.537865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537808 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:03.538068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537873 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.538068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.537899 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo\") pod \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\" (UID: \"f7ecbe05-9f48-4b2a-8654-0823907a23aa\") " Nov 02 03:36:03.538526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.538275 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.538526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.538296 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:36:03.538526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.538326 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:03.538526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.538401 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:03.538800 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.538626 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:03.539862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.539824 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:36:03.540112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.540093 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:03.540195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.540177 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "kube-api-access-mrk4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:36:03.540319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.540302 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo" pod "f7ecbe05-9f48-4b2a-8654-0823907a23aa" (UID: "f7ecbe05-9f48-4b2a-8654-0823907a23aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:36:03.638843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638812 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.638843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638834 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.638843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638844 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.639040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638854 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.639040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638863 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.639040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638872 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrk4n\" (UniqueName: \"kubernetes.io/projected/f7ecbe05-9f48-4b2a-8654-0823907a23aa-kube-api-access-mrk4n\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.639040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638881 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ecbe05-9f48-4b2a-8654-0823907a23aa-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.639040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.638889 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ecbe05-9f48-4b2a-8654-0823907a23aa-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:03.975777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.975732 4868 generic.go:356] "Generic (PLEG): container finished" podID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerID="f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68" exitCode=0 Nov 02 03:36:03.976197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.975803 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" Nov 02 03:36:03.976197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.975813 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" event={"ID":"f7ecbe05-9f48-4b2a-8654-0823907a23aa","Type":"ContainerDied","Data":"f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68"} Nov 02 03:36:03.976197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.975898 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9" event={"ID":"f7ecbe05-9f48-4b2a-8654-0823907a23aa","Type":"ContainerDied","Data":"a01dc3847ee76f35fc88d7e6a068128659a40db7a1dcaa6158b365c43988e307"} Nov 02 03:36:03.976197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.975927 4868 scope.go:122] "RemoveContainer" containerID="f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68" Nov 02 03:36:03.986934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.986911 4868 scope.go:122] "RemoveContainer" containerID="2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478" Nov 02 03:36:03.995904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.995882 4868 scope.go:122] "RemoveContainer" containerID="f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68" Nov 02 03:36:03.996146 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:03.996127 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68\": container with ID starting with f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68 not found: ID does not exist" containerID="f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68" Nov 02 03:36:03.996213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.996154 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68"} err="failed to get container status \"f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68\": rpc error: code = NotFound desc = could not find container \"f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68\": container with ID starting with f41c6b0eb4269fd2da5fe77744a5c89a8ae5038b5c6f3424d14bd22b056b9e68 not found: ID does not exist" Nov 02 03:36:03.996213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.996171 4868 scope.go:122] "RemoveContainer" containerID="2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478" Nov 02 03:36:03.996430 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:03.996409 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478\": container with ID starting with 2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478 not found: ID does not exist" containerID="2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478" Nov 02 03:36:03.996526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:03.996434 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478"} err="failed to get container status \"2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478\": rpc error: code = NotFound desc = could not find container \"2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478\": container with ID starting with 2fff5258cc034833b1697d83b64330945aaf44b59ec27a746f23bb2fb15b0478 not found: ID does not exist" Nov 02 03:36:04.139677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:04.139634 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9"] Nov 02 03:36:04.189138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:04.189111 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f1a692-openshift-default-5d6d98688f-wrrc9"] Nov 02 03:36:06.114833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:06.114802 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" path="/var/lib/kubelet/pods/f7ecbe05-9f48-4b2a-8654-0823907a23aa/volumes" Nov 02 03:36:06.662738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:06.662704 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs"] Nov 02 03:36:06.663234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:06.663203 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" containerID="cri-o://ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5" gracePeriod=30 Nov 02 03:36:11.926066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:11.926041 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:36:12.023491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.023424 4868 generic.go:356] "Generic (PLEG): container finished" podID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerID="ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5" exitCode=0 Nov 02 03:36:12.023491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.023485 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" Nov 02 03:36:12.023703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.023509 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" event={"ID":"a9a68900-05c8-4e9c-b89d-ef751561e4f7","Type":"ContainerDied","Data":"ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5"} Nov 02 03:36:12.023703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.023583 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs" event={"ID":"a9a68900-05c8-4e9c-b89d-ef751561e4f7","Type":"ContainerDied","Data":"603e610dae086d53e24832928686d68a21fa146cca9c915de99586450a711e24"} Nov 02 03:36:12.023703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.023616 4868 scope.go:122] "RemoveContainer" containerID="ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5" Nov 02 03:36:12.032910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.032894 4868 scope.go:122] "RemoveContainer" containerID="f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8" Nov 02 03:36:12.041473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.041459 4868 scope.go:122] "RemoveContainer" containerID="ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5" Nov 02 03:36:12.041719 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:12.041701 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5\": container with ID starting with ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5 not found: ID does not exist" containerID="ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5" Nov 02 03:36:12.041789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.041727 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5"} err="failed to get container status \"ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5\": rpc error: code = NotFound desc = could not find container \"ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5\": container with ID starting with ef2e9827039838a5d86d39b498560233431ee25753036debf1a2554d4c6741b5 not found: ID does not exist" Nov 02 03:36:12.041789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.041749 4868 scope.go:122] "RemoveContainer" containerID="f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8" Nov 02 03:36:12.042005 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:12.041990 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8\": container with ID starting with f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8 not found: ID does not exist" containerID="f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8" Nov 02 03:36:12.042057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.042008 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8"} err="failed to get container status \"f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8\": rpc error: code = NotFound desc = could not find container \"f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8\": container with ID starting with f7fb794fd22ccb3f8de6bbae5e3740a4f20f0c0ddc2b5fb3727555798f7820d8 not found: ID does not exist" Nov 02 03:36:12.047400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047380 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047425 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047500 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047529 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047568 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047685 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:12.047768 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047691 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047781 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:12.047812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047790 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047876 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.047916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047895 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:36:12.047916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.047907 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo\") pod \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\" (UID: \"a9a68900-05c8-4e9c-b89d-ef751561e4f7\") " Nov 02 03:36:12.048093 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048062 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:12.048178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048160 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:12.048423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048403 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.048423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048422 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.048610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048430 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.048610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048439 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.048610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.048450 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.049834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.049810 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:36:12.049963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.049944 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:12.050023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.049979 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "kube-api-access-gwx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:36:12.050023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.049999 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo" pod "a9a68900-05c8-4e9c-b89d-ef751561e4f7" (UID: "a9a68900-05c8-4e9c-b89d-ef751561e4f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:36:12.149570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.149542 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwx7d\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-kube-api-access-gwx7d\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.149570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.149568 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.149717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.149577 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.149717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.149586 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a68900-05c8-4e9c-b89d-ef751561e4f7-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:12.436387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.436358 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs"] Nov 02 03:36:12.439668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:12.439638 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-abc348-openshift-default-7bd4495477-kntqs"] Nov 02 03:36:14.115683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:14.115643 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" path="/var/lib/kubelet/pods/a9a68900-05c8-4e9c-b89d-ef751561e4f7/volumes" Nov 02 03:36:33.445574 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:33.445539 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:36:45.698821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:45.698787 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps"] Nov 02 03:36:45.699208 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:45.699022 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" containerID="cri-o://64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d" gracePeriod=30 Nov 02 03:36:48.234886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.234853 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm"] Nov 02 03:36:48.235484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235465 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" Nov 02 03:36:48.235542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235487 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" Nov 02 03:36:48.235542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235508 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" Nov 02 03:36:48.235542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235515 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" Nov 02 03:36:48.235676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235662 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" Nov 02 03:36:48.235712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235686 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" Nov 02 03:36:48.235712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235705 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" Nov 02 03:36:48.235782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.235719 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" Nov 02 03:36:48.240533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.240516 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.294589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.294550 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm"] Nov 02 03:36:48.322139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.322116 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-e44998-serving-cert\"" Nov 02 03:36:48.330337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.330307 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.330493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.330470 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.330644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.330625 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnnx\" (UniqueName: \"kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.330740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.330673 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.431810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.431779 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.431941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.431828 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.431941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.431895 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.432054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.431989 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnnx\" (UniqueName: \"kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.432109 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:48.432075 4868 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-serving-cert: secret "maas-api-e2e-praxis-a-e44998-serving-cert" not found Nov 02 03:36:48.432178 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:48.432163 4868 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls podName:9954f9a7-7914-4d15-b9f3-2dd428b3c2cb nodeName:}" failed. No retries permitted until 2026-11-02 03:36:48.932142504 +0000 UTC m=+2950.927843438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls") pod "maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" (UID: "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb") : secret "maas-api-e2e-praxis-a-e44998-serving-cert" not found Nov 02 03:36:48.432489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.432473 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.434180 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.434160 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.510154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.508981 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnnx\" (UniqueName: \"kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.938017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.937983 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:48.940526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:48.940500 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e44998-84979bf95b-782jm\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:49.108149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:49.108116 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd"] Nov 02 03:36:49.153248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:49.153217 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:49.523671 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:49.523650 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm"] Nov 02 03:36:49.526466 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:36:49.526436 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9954f9a7_7914_4d15_b9f3_2dd428b3c2cb.slice/crio-fd6be8ce1c5cd193ad7a049e6f21f96278c1937e0e0abe39e72f90b1020a72d5 WatchSource:0}: Error finding container fd6be8ce1c5cd193ad7a049e6f21f96278c1937e0e0abe39e72f90b1020a72d5: Status 404 returned error can't find the container with id fd6be8ce1c5cd193ad7a049e6f21f96278c1937e0e0abe39e72f90b1020a72d5 Nov 02 03:36:49.527695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:49.527673 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:36:50.251075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:50.251032 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" event={"ID":"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb","Type":"ContainerStarted","Data":"fd6be8ce1c5cd193ad7a049e6f21f96278c1937e0e0abe39e72f90b1020a72d5"} Nov 02 03:36:50.963920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:50.963893 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:36:51.064703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064631 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.064855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064716 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.064855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064755 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.064855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064811 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.064855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064840 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.065076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064870 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.065076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064958 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.065076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.064999 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.065076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065042 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket\") pod \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\" (UID: \"4fb9009a-ce66-424c-ab6f-26bd8e115b29\") " Nov 02 03:36:51.065297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065155 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:51.065297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065066 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:36:51.065452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065337 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:51.065452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065393 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:51.065574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065557 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.065633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065580 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.065633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065597 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.065633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065611 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.065633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.065560 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:51.068001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.067973 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:36:51.068001 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.067987 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:36:51.068142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.068003 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:36:51.068142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.068012 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545" pod "4fb9009a-ce66-424c-ab6f-26bd8e115b29" (UID: "4fb9009a-ce66-424c-ab6f-26bd8e115b29"). InnerVolumeSpecName "kube-api-access-qp545". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:36:51.167079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.167047 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp545\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-kube-api-access-qp545\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.167079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.167075 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.167079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.167085 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.167294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.167094 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.167294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.167102 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb9009a-ce66-424c-ab6f-26bd8e115b29-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:36:51.286639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.286603 4868 generic.go:356] "Generic (PLEG): container finished" podID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerID="64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d" exitCode=0 Nov 02 03:36:51.286951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.286924 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" event={"ID":"4fb9009a-ce66-424c-ab6f-26bd8e115b29","Type":"ContainerDied","Data":"64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d"} Nov 02 03:36:51.287125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.287110 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" event={"ID":"4fb9009a-ce66-424c-ab6f-26bd8e115b29","Type":"ContainerDied","Data":"c1d0356ec0085f5a415af32bbd95517ca62b9019ec084fb4eb2efd6a0592c771"} Nov 02 03:36:51.287242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.287230 4868 scope.go:122] "RemoveContainer" containerID="64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d" Nov 02 03:36:51.287436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.287015 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps" Nov 02 03:36:51.291487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.291452 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" event={"ID":"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb","Type":"ContainerStarted","Data":"d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c"} Nov 02 03:36:51.291807 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.291782 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:36:51.302036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.302019 4868 scope.go:122] "RemoveContainer" containerID="64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d" Nov 02 03:36:51.302337 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:36:51.302316 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d\": container with ID starting with 64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d not found: ID does not exist" containerID="64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d" Nov 02 03:36:51.302419 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.302345 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d"} err="failed to get container status \"64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d\": rpc error: code = NotFound desc = could not find container \"64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d\": container with ID starting with 64e94d13e39cf3a1e43dcaf8a617d296ee43f32cd5d7d5a793ad1e4b5ffd794d not found: ID does not exist" Nov 02 03:36:51.453403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.453344 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" podStartSLOduration=2.587392974 podStartE2EDuration="3.453329786s" totalImagesPullingTime="865.936812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:36:51.451897888 +0000 UTC m=+2953.447598835" watchObservedRunningTime="2026-11-02 03:36:51.453329786 +0000 UTC m=+2953.449030729" Nov 02 03:36:51.563144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.563108 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps"] Nov 02 03:36:51.602804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:51.599942 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-7lxps"] Nov 02 03:36:52.115234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:52.115204 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" path="/var/lib/kubelet/pods/4fb9009a-ce66-424c-ab6f-26bd8e115b29/volumes" Nov 02 03:36:57.304091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:36:57.304063 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:37:03.445648 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:37:03.445609 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:37:04.128985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.128950 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" podUID="add71940-ba82-4b77-8e3d-ff5efe0a5b71" containerName="main" containerID="cri-o://0fba4fe31423fa1223761902e62de9ad312c771c72519865839f15879286dc99" gracePeriod=45 Nov 02 03:37:04.365048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.365019 4868 generic.go:356] "Generic (PLEG): container finished" podID="add71940-ba82-4b77-8e3d-ff5efe0a5b71" containerID="0fba4fe31423fa1223761902e62de9ad312c771c72519865839f15879286dc99" exitCode=0 Nov 02 03:37:04.365219 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.365096 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" event={"ID":"add71940-ba82-4b77-8e3d-ff5efe0a5b71","Type":"ContainerDied","Data":"0fba4fe31423fa1223761902e62de9ad312c771c72519865839f15879286dc99"} Nov 02 03:37:04.388445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.388395 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.513299 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache\") pod \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.513416 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home\") pod \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.513545 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir\") pod \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.513583 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b\" (UniqueName: \"kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b\") pod \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.513613 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm\") pod \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.513656 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs\" (UniqueName: \"kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs\") pod \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\" (UID: \"add71940-ba82-4b77-8e3d-ff5efe0a5b71\") " Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.514060 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir" pod "add71940-ba82-4b77-8e3d-ff5efe0a5b71" (UID: "add71940-ba82-4b77-8e3d-ff5efe0a5b71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.514289 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache" pod "add71940-ba82-4b77-8e3d-ff5efe0a5b71" (UID: "add71940-ba82-4b77-8e3d-ff5efe0a5b71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:04.513339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.514498 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home" pod "add71940-ba82-4b77-8e3d-ff5efe0a5b71" (UID: "add71940-ba82-4b77-8e3d-ff5efe0a5b71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:04.516888 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.516853 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs" pod "add71940-ba82-4b77-8e3d-ff5efe0a5b71" (UID: "add71940-ba82-4b77-8e3d-ff5efe0a5b71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:37:04.521647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.521625 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm" pod "add71940-ba82-4b77-8e3d-ff5efe0a5b71" (UID: "add71940-ba82-4b77-8e3d-ff5efe0a5b71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:04.526201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.526179 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b" pod "add71940-ba82-4b77-8e3d-ff5efe0a5b71" (UID: "add71940-ba82-4b77-8e3d-ff5efe0a5b71"). InnerVolumeSpecName "kube-api-access-zdn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:37:04.615226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.615188 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:04.615226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.615224 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdn8b\" (UniqueName: \"kubernetes.io/projected/add71940-ba82-4b77-8e3d-ff5efe0a5b71-kube-api-access-zdn8b\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:04.615511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.615239 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:04.615511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.615252 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add71940-ba82-4b77-8e3d-ff5efe0a5b71-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:04.615511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.615265 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:04.615511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:04.615278 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add71940-ba82-4b77-8e3d-ff5efe0a5b71-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:05.371867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:05.371837 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" Nov 02 03:37:05.372070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:05.371873 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd" event={"ID":"add71940-ba82-4b77-8e3d-ff5efe0a5b71","Type":"ContainerDied","Data":"5659c0731e70346ca00ca8fcfa50c309a631fc812b11bcf45aab2e5c3d6b26cc"} Nov 02 03:37:05.372070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:05.371923 4868 scope.go:122] "RemoveContainer" containerID="0fba4fe31423fa1223761902e62de9ad312c771c72519865839f15879286dc99" Nov 02 03:37:05.534561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:05.534531 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd"] Nov 02 03:37:05.583874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:05.583846 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-dd87f0/ipp-route-dd87f0-kserve-54d77955c-znbrd"] Nov 02 03:37:06.114609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:06.114575 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add71940-ba82-4b77-8e3d-ff5efe0a5b71" path="/var/lib/kubelet/pods/add71940-ba82-4b77-8e3d-ff5efe0a5b71/volumes" Nov 02 03:37:22.662833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:22.662803 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf"] Nov 02 03:37:22.663405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:22.663036 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" containerID="cri-o://54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8" gracePeriod=30 Nov 02 03:37:24.555271 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555238 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk"] Nov 02 03:37:24.555725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555711 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" Nov 02 03:37:24.555770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555725 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a68900-05c8-4e9c-b89d-ef751561e4f7" containerName="istio-proxy" Nov 02 03:37:24.555770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555737 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" Nov 02 03:37:24.555770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555742 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ecbe05-9f48-4b2a-8654-0823907a23aa" containerName="istio-proxy" Nov 02 03:37:24.555770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555757 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" Nov 02 03:37:24.555770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555763 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" Nov 02 03:37:24.555925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555780 4868 cpu_manager.go:414] "containerMap: removing container" podUID="add71940-ba82-4b77-8e3d-ff5efe0a5b71" containerName="main" Nov 02 03:37:24.555925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555785 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add71940-ba82-4b77-8e3d-ff5efe0a5b71" containerName="main" Nov 02 03:37:24.555925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555874 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="add71940-ba82-4b77-8e3d-ff5efe0a5b71" containerName="main" Nov 02 03:37:24.555925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.555883 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb9009a-ce66-424c-ab6f-26bd8e115b29" containerName="istio-proxy" Nov 02 03:37:24.559379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.559360 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.643953 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644032 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfnv\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644141 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644195 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644241 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644291 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644315 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644377 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.644482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.644410 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.667660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.667619 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk"] Nov 02 03:37:24.670259 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.670239 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-167a01-openshift-default-dockercfg-ssmv5\"" Nov 02 03:37:24.745878 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.745839 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.745925 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.745983 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746017 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746041 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746092 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746125 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746257 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746160 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746257 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746214 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfnv\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.746887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.746779 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.750097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.747340 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.750097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.749576 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.750097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.750052 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.753376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.753304 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.753711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.753693 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.755886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.755856 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.805074 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.805051 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.805417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.805344 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfnv\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv\") pod \"e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:24.872754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:24.872723 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:25.296780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.296751 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk"] Nov 02 03:37:25.298865 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:37:25.298837 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f3df67_086d_46a5_85cf_fbc65b88b874.slice/crio-dbe5c1ccab65d64b404938ae7298ffecbde500f10624411cca22de3b5114af6a WatchSource:0}: Error finding container dbe5c1ccab65d64b404938ae7298ffecbde500f10624411cca22de3b5114af6a: Status 404 returned error can't find the container with id dbe5c1ccab65d64b404938ae7298ffecbde500f10624411cca22de3b5114af6a Nov 02 03:37:25.301285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.301251 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:37:25.301387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.301316 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:37:25.301387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.301345 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:37:25.491002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.490961 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" event={"ID":"28f3df67-086d-46a5-85cf-fbc65b88b874","Type":"ContainerStarted","Data":"224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e"} Nov 02 03:37:25.491192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.491069 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" event={"ID":"28f3df67-086d-46a5-85cf-fbc65b88b874","Type":"ContainerStarted","Data":"dbe5c1ccab65d64b404938ae7298ffecbde500f10624411cca22de3b5114af6a"} Nov 02 03:37:25.629380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.628318 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podStartSLOduration=1.62829859 podStartE2EDuration="1.62829859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:37:25.625547237 +0000 UTC m=+2987.621248179" watchObservedRunningTime="2026-11-02 03:37:25.62829859 +0000 UTC m=+2987.623999536" Nov 02 03:37:25.873845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:25.873582 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:26.878313 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:26.878281 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:27.501602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:27.501569 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:27.502631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:27.502606 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:37:27.930109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:27.930086 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:37:28.087452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087379 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.087452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087429 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.087720 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087463 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.087720 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087646 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.087720 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087691 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.087889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087727 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:28.087889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087737 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.087889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087814 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.088048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087917 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.088048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087958 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data\") pod \"e6857c35-deec-4921-867b-e4e21ee15205\" (UID: \"e6857c35-deec-4921-867b-e4e21ee15205\") " Nov 02 03:37:28.088048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.087978 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:28.088048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088002 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:28.088246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088228 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:28.088303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088276 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:37:28.088469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088444 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6857c35-deec-4921-867b-e4e21ee15205-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.088469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088470 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.088636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088487 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.088636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088501 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.088636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.088516 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.089999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.089973 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:37:28.090080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.089997 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:37:28.090213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.090192 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "kube-api-access-zvv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:37:28.090213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.090204 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token" pod "e6857c35-deec-4921-867b-e4e21ee15205" (UID: "e6857c35-deec-4921-867b-e4e21ee15205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:37:28.189561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.189532 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6857c35-deec-4921-867b-e4e21ee15205-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.189561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.189560 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.189766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.189573 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6857c35-deec-4921-867b-e4e21ee15205-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.189766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.189587 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvv9r\" (UniqueName: \"kubernetes.io/projected/e6857c35-deec-4921-867b-e4e21ee15205-kube-api-access-zvv9r\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:37:28.508454 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.508422 4868 generic.go:356] "Generic (PLEG): container finished" podID="e6857c35-deec-4921-867b-e4e21ee15205" containerID="54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8" exitCode=0 Nov 02 03:37:28.508649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.508480 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" Nov 02 03:37:28.508649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.508517 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" event={"ID":"e6857c35-deec-4921-867b-e4e21ee15205","Type":"ContainerDied","Data":"54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8"} Nov 02 03:37:28.508649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.508601 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf" event={"ID":"e6857c35-deec-4921-867b-e4e21ee15205","Type":"ContainerDied","Data":"470f47fba0ad566d005a271e67bebe261c619ac3601f0cd4d650e1c2723b4038"} Nov 02 03:37:28.508792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.508651 4868 scope.go:122] "RemoveContainer" containerID="54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8" Nov 02 03:37:28.519266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.519251 4868 scope.go:122] "RemoveContainer" containerID="2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f" Nov 02 03:37:28.528619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.528599 4868 scope.go:122] "RemoveContainer" containerID="54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8" Nov 02 03:37:28.528888 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:37:28.528868 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8\": container with ID starting with 54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8 not found: ID does not exist" containerID="54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8" Nov 02 03:37:28.528946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.528897 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8"} err="failed to get container status \"54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8\": rpc error: code = NotFound desc = could not find container \"54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8\": container with ID starting with 54c6e04c8f052f418061a0c12c13b3c089d73455cd7ee233a835de2bd0e22dc8 not found: ID does not exist" Nov 02 03:37:28.528946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.528922 4868 scope.go:122] "RemoveContainer" containerID="2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f" Nov 02 03:37:28.529164 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:37:28.529137 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f\": container with ID starting with 2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f not found: ID does not exist" containerID="2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f" Nov 02 03:37:28.529211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.529171 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f"} err="failed to get container status \"2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f\": rpc error: code = NotFound desc = could not find container \"2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f\": container with ID starting with 2480e7313292b244db6cebe930b6dc4d62c857618202fce5c365eb86a31c504f not found: ID does not exist" Nov 02 03:37:28.723058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.723026 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf"] Nov 02 03:37:28.772189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:28.772115 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0809c8-openshift-default-7dbbd95d5c-xlgwf"] Nov 02 03:37:30.114747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:30.114717 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6857c35-deec-4921-867b-e4e21ee15205" path="/var/lib/kubelet/pods/e6857c35-deec-4921-867b-e4e21ee15205/volumes" Nov 02 03:37:33.445338 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:37:33.445303 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:37:38.474341 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:38.474315 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:37:38.479520 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:38.479498 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:37:57.688746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:57.688717 4868 generic.go:356] "Generic (PLEG): container finished" podID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerID="4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461" exitCode=0 Nov 02 03:37:57.689102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:57.688791 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" event={"ID":"e35fd55b-dbd5-4fa0-9782-fd36825c9850","Type":"ContainerDied","Data":"4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461"} Nov 02 03:37:57.689222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:57.689206 4868 scope.go:122] "RemoveContainer" containerID="4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461" Nov 02 03:37:57.690738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:57.690714 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:37:57.690806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:57.690778 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:37:57.690846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:57.690806 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:37:58.697047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:58.697018 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" event={"ID":"e35fd55b-dbd5-4fa0-9782-fd36825c9850","Type":"ContainerStarted","Data":"38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250"} Nov 02 03:37:58.753842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:58.752709 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:37:58.756778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:58.756494 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 03:37:58.756778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:58.756547 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 03:37:59.751375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:59.742652 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 03:37:59.751375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:37:59.742712 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 03:38:00.742777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:00.742742 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 03:38:00.743052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:00.743022 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 03:38:01.739868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:01.739829 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 03:38:01.740493 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:01.740457 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 03:38:02.740756 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:02.740712 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 03:38:02.741276 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:02.740778 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 03:38:02.963544 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:02.963501 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:38:03.469967 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:38:03.469925 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:38:03.740541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:03.740468 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 03:38:03.740541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:03.740518 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 03:38:04.744545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:04.744510 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:38:04.745532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:04.745515 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:38:27.281497 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:27.281455 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th"] Nov 02 03:38:28.860861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.860825 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc"] Nov 02 03:38:28.861272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.861259 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" Nov 02 03:38:28.861317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.861273 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" Nov 02 03:38:28.861393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.861383 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" Nov 02 03:38:28.861434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.861395 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" Nov 02 03:38:28.864670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.864651 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.903326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.903284 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.904120 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.904093 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvpr\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.904396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.904373 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.904639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.904618 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.904925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.904905 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.905240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.905190 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.905591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.905567 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.905942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.905921 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.906137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.906112 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:28.956398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:28.955723 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1b1008-openshift-default-dockercfg-b2h2n\"" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.004055 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc"] Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.041853 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.041907 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvpr\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.041946 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.041975 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.042043 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.042138 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.042203 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.042235 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.045379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.042260 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.051237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.051204 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.052505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.051468 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.052505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.051756 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.052505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.051777 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.052505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.052017 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.053670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.053646 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.061625 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.061601 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.119733 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.119663 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.131916 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.131884 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvpr\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr\") pod \"e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.177950 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.177911 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:29.602956 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.602929 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc"] Nov 02 03:38:29.605104 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:38:29.605075 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc327185_945b_4338_ae87_1c38841a752f.slice/crio-0c7318a290d1be87c84f15d704f5f6ec65e89478a9d9c6290200773a63261425 WatchSource:0}: Error finding container 0c7318a290d1be87c84f15d704f5f6ec65e89478a9d9c6290200773a63261425: Status 404 returned error can't find the container with id 0c7318a290d1be87c84f15d704f5f6ec65e89478a9d9c6290200773a63261425 Nov 02 03:38:29.607903 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.607868 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:38:29.607990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.607950 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:38:29.608029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.607994 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:38:29.883140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.883039 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" event={"ID":"dc327185-945b-4338-ae87-1c38841a752f","Type":"ContainerStarted","Data":"8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893"} Nov 02 03:38:29.883140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:29.883100 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" event={"ID":"dc327185-945b-4338-ae87-1c38841a752f","Type":"ContainerStarted","Data":"0c7318a290d1be87c84f15d704f5f6ec65e89478a9d9c6290200773a63261425"} Nov 02 03:38:30.002087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:30.001868 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podStartSLOduration=2.001852109 podStartE2EDuration="2.001852109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:38:30.000398849 +0000 UTC m=+3051.996099822" watchObservedRunningTime="2026-11-02 03:38:30.001852109 +0000 UTC m=+3051.997553052" Nov 02 03:38:30.178927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:30.178837 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:30.185405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:30.185367 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:30.894270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:30.892342 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:30.894270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:30.893446 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:38:33.447852 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:38:33.447811 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:38:42.301295 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.301253 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" podUID="e1e6c082-411e-42c1-972f-17d3b7c0a1b0" containerName="main" containerID="cri-o://af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636" gracePeriod=45 Nov 02 03:38:42.569317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.569246 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:38:42.693024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.692992 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir\") pod \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " Nov 02 03:38:42.693195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693033 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs\") pod \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " Nov 02 03:38:42.693195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693124 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm\") pod \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " Nov 02 03:38:42.693195 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693188 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache\") pod \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " Nov 02 03:38:42.693429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693243 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm\" (UniqueName: \"kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm\") pod \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " Nov 02 03:38:42.693429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693285 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir" pod "e1e6c082-411e-42c1-972f-17d3b7c0a1b0" (UID: "e1e6c082-411e-42c1-972f-17d3b7c0a1b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:38:42.693429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693297 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home\") pod \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\" (UID: \"e1e6c082-411e-42c1-972f-17d3b7c0a1b0\") " Nov 02 03:38:42.693606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693492 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache" pod "e1e6c082-411e-42c1-972f-17d3b7c0a1b0" (UID: "e1e6c082-411e-42c1-972f-17d3b7c0a1b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:38:42.693689 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693669 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home" pod "e1e6c082-411e-42c1-972f-17d3b7c0a1b0" (UID: "e1e6c082-411e-42c1-972f-17d3b7c0a1b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:38:42.693864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693811 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:38:42.693949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.693873 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:38:42.695980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.695942 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm" pod "e1e6c082-411e-42c1-972f-17d3b7c0a1b0" (UID: "e1e6c082-411e-42c1-972f-17d3b7c0a1b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:38:42.695980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.695971 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm" pod "e1e6c082-411e-42c1-972f-17d3b7c0a1b0" (UID: "e1e6c082-411e-42c1-972f-17d3b7c0a1b0"). InnerVolumeSpecName "kube-api-access-njkwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:38:42.696387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.696346 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs" pod "e1e6c082-411e-42c1-972f-17d3b7c0a1b0" (UID: "e1e6c082-411e-42c1-972f-17d3b7c0a1b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:38:42.795284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.795240 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:38:42.795284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.795272 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:38:42.795475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.795288 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njkwm\" (UniqueName: \"kubernetes.io/projected/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-kube-api-access-njkwm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:38:42.795475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.795302 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1e6c082-411e-42c1-972f-17d3b7c0a1b0-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:38:42.956293 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.956261 4868 generic.go:356] "Generic (PLEG): container finished" podID="e1e6c082-411e-42c1-972f-17d3b7c0a1b0" containerID="af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636" exitCode=0 Nov 02 03:38:42.956498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.956308 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" Nov 02 03:38:42.956498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.956346 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" event={"ID":"e1e6c082-411e-42c1-972f-17d3b7c0a1b0","Type":"ContainerDied","Data":"af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636"} Nov 02 03:38:42.956498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.956406 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th" event={"ID":"e1e6c082-411e-42c1-972f-17d3b7c0a1b0","Type":"ContainerDied","Data":"a264964cb1e49a9a530bbc264e6619d1d1574d6100cd4862100bd70cf91cd0f2"} Nov 02 03:38:42.956498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.956430 4868 scope.go:122] "RemoveContainer" containerID="af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636" Nov 02 03:38:42.966799 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.966775 4868 scope.go:122] "RemoveContainer" containerID="af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636" Nov 02 03:38:42.967121 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:38:42.967098 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636\": container with ID starting with af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636 not found: ID does not exist" containerID="af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636" Nov 02 03:38:42.967191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:42.967129 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636"} err="failed to get container status \"af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636\": rpc error: code = NotFound desc = could not find container \"af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636\": container with ID starting with af81ace2b3518e8d4e4434b4f6fdb34f28a4fed836d0f4616182fc0035c02636 not found: ID does not exist" Nov 02 03:38:43.119080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:43.119046 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th"] Nov 02 03:38:43.123274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:43.123250 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d94cc3/tenantcfg-model-d94cc3-kserve-5f9bcd5cb9-zc2th"] Nov 02 03:38:44.114877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:38:44.114848 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e6c082-411e-42c1-972f-17d3b7c0a1b0" path="/var/lib/kubelet/pods/e1e6c082-411e-42c1-972f-17d3b7c0a1b0/volumes" Nov 02 03:39:03.445902 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:39:03.445864 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:39:12.672585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:12.671514 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": read tcp 10.132.0.2:33602->10.132.0.113:15021: read: connection reset by peer" start-of-body= Nov 02 03:39:12.672585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:12.671579 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": read tcp 10.132.0.2:33602->10.132.0.113:15021: read: connection reset by peer" Nov 02 03:39:13.182089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:13.182056 4868 generic.go:356] "Generic (PLEG): container finished" podID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerID="224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e" exitCode=0 Nov 02 03:39:13.182291 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:13.182096 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" event={"ID":"28f3df67-086d-46a5-85cf-fbc65b88b874","Type":"ContainerDied","Data":"224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e"} Nov 02 03:39:13.182525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:13.182507 4868 scope.go:122] "RemoveContainer" containerID="224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e" Nov 02 03:39:13.184152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:13.184123 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:39:13.184242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:13.184196 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:39:13.184242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:13.184232 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:39:14.190184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:14.190153 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" event={"ID":"28f3df67-086d-46a5-85cf-fbc65b88b874","Type":"ContainerStarted","Data":"3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8"} Nov 02 03:39:14.873330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:14.873294 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:39:14.873531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:14.873467 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 03:39:14.873531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:14.873519 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 03:39:15.873858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:15.873821 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 03:39:15.874394 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:15.873880 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 03:39:16.873667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:16.873628 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 03:39:16.873880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:16.873707 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 03:39:17.873491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:17.873457 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 03:39:17.873672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:17.873530 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 03:39:18.874102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:18.874023 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 03:39:18.874102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:18.874077 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 03:39:19.877521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:19.877488 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:39:19.877926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:19.877830 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:39:19.878559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:19.878538 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:39:33.446397 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:39:33.446338 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:39:51.865453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.865418 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4"] Nov 02 03:39:51.865938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.865923 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" Nov 02 03:39:51.865986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.865939 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6857c35-deec-4921-867b-e4e21ee15205" containerName="istio-proxy" Nov 02 03:39:51.865986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.865969 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e1e6c082-411e-42c1-972f-17d3b7c0a1b0" containerName="main" Nov 02 03:39:51.865986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.865974 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e6c082-411e-42c1-972f-17d3b7c0a1b0" containerName="main" Nov 02 03:39:51.866079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.866060 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e6c082-411e-42c1-972f-17d3b7c0a1b0" containerName="main" Nov 02 03:39:51.870808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.870788 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:51.924599 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.924569 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4"] Nov 02 03:39:51.979373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.971687 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9b06f8-dockercfg-vbhb4\"" Nov 02 03:39:51.979373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.971790 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9b06f8\"" Nov 02 03:39:51.992420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.987571 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:51.992420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:51.987718 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwcw\" (UniqueName: \"kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw\") pod \"payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:52.025520 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.025485 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d"] Nov 02 03:39:52.035014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.034987 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.081237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.081203 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d"] Nov 02 03:39:52.088527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.088490 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwcw\" (UniqueName: \"kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw\") pod \"payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:52.088672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.088544 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.088672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.088572 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dng7g\" (UniqueName: \"kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g\") pod \"payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.088887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.088780 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:52.089749 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.089715 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:52.145132 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.145073 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwcw\" (UniqueName: \"kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw\") pod \"payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:52.184658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.184081 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:52.190212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.190188 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.190324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.190219 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dng7g\" (UniqueName: \"kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g\") pod \"payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.190938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.190918 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.247517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.247489 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dng7g\" (UniqueName: \"kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g\") pod \"payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.350753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.350725 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:52.554603 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.554574 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4"] Nov 02 03:39:52.555909 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:39:52.555883 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dd5c16_1ef7_4965_84ff_441bfb03c477.slice/crio-8a5a5bcc5b8974fcd34abfc48f5a75f2365a507ca693e5e5e8d7ec739da018f8 WatchSource:0}: Error finding container 8a5a5bcc5b8974fcd34abfc48f5a75f2365a507ca693e5e5e8d7ec739da018f8: Status 404 returned error can't find the container with id 8a5a5bcc5b8974fcd34abfc48f5a75f2365a507ca693e5e5e8d7ec739da018f8 Nov 02 03:39:52.720630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:52.720607 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d"] Nov 02 03:39:52.721982 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:39:52.721962 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa218932_b8b9_4add_9ecc_304a2516fc22.slice/crio-86c0717e1b208f16900cd96c06cd10c3582a783ee7bb2e53086c1cfd4f46dc53 WatchSource:0}: Error finding container 86c0717e1b208f16900cd96c06cd10c3582a783ee7bb2e53086c1cfd4f46dc53: Status 404 returned error can't find the container with id 86c0717e1b208f16900cd96c06cd10c3582a783ee7bb2e53086c1cfd4f46dc53 Nov 02 03:39:53.466628 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.466591 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" event={"ID":"aa218932-b8b9-4add-9ecc-304a2516fc22","Type":"ContainerStarted","Data":"682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad"} Nov 02 03:39:53.467129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.466648 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" event={"ID":"aa218932-b8b9-4add-9ecc-304a2516fc22","Type":"ContainerStarted","Data":"86c0717e1b208f16900cd96c06cd10c3582a783ee7bb2e53086c1cfd4f46dc53"} Nov 02 03:39:53.467129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.466676 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:39:53.468213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.468191 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" event={"ID":"08dd5c16-1ef7-4965-84ff-441bfb03c477","Type":"ContainerStarted","Data":"42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853"} Nov 02 03:39:53.468329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.468227 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" event={"ID":"08dd5c16-1ef7-4965-84ff-441bfb03c477","Type":"ContainerStarted","Data":"8a5a5bcc5b8974fcd34abfc48f5a75f2365a507ca693e5e5e8d7ec739da018f8"} Nov 02 03:39:53.468329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.468256 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:39:53.580635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.580591 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" podStartSLOduration=2.5805701340000002 podStartE2EDuration="2.580570134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:39:53.578095893 +0000 UTC m=+3135.573796840" watchObservedRunningTime="2026-11-02 03:39:53.580570134 +0000 UTC m=+3135.576271125" Nov 02 03:39:53.692118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:39:53.692063 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" podStartSLOduration=2.692046899 podStartE2EDuration="2.692046899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:39:53.689924757 +0000 UTC m=+3135.685625714" watchObservedRunningTime="2026-11-02 03:39:53.692046899 +0000 UTC m=+3135.687747844" Nov 02 03:40:03.445848 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:40:03.445810 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:40:04.476614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:04.476587 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:40:04.477057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:04.476658 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:40:15.609975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.609945 4868 generic.go:356] "Generic (PLEG): container finished" podID="dc327185-945b-4338-ae87-1c38841a752f" containerID="8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893" exitCode=0 Nov 02 03:40:15.610408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.610020 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" event={"ID":"dc327185-945b-4338-ae87-1c38841a752f","Type":"ContainerDied","Data":"8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893"} Nov 02 03:40:15.610408 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.610382 4868 scope.go:122] "RemoveContainer" containerID="8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893" Nov 02 03:40:15.611906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.611878 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:40:15.612002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.611944 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:40:15.612002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.611975 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:40:15.892690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:15.892605 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:40:16.617791 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:16.617755 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" event={"ID":"dc327185-945b-4338-ae87-1c38841a752f","Type":"ContainerStarted","Data":"b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14"} Nov 02 03:40:17.179025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:17.178987 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:40:17.180302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:17.180275 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 03:40:17.180447 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:17.180322 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 03:40:18.178627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:18.178596 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 03:40:18.179009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:18.178657 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 03:40:19.178895 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:19.178858 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 03:40:19.179282 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:19.178933 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 03:40:20.178883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:20.178847 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 03:40:20.179102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:20.178919 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 03:40:21.178575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:21.178542 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 03:40:21.178751 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:21.178593 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 03:40:22.184745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:22.184716 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:40:22.185176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:22.185030 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:40:22.185823 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:22.185800 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:40:26.872851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.872817 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b"] Nov 02 03:40:26.879805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.879767 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:26.932587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.932559 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b"] Nov 02 03:40:26.965563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.965535 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:26.965734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.965687 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:26.965734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.965708 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528l9\" (UniqueName: \"kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:26.979072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.979042 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:40:26.979255 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:26.979234 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:40:27.067581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.067541 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.067784 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.067589 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528l9\" (UniqueName: \"kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.067784 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.067662 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.068965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.068936 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.082387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.076089 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.124921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.124830 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-528l9\" (UniqueName: \"kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9\") pod \"maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.193346 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.193314 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:27.565019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.564986 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b"] Nov 02 03:40:27.566735 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:40:27.566709 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d9395d_42a0_49fd_a95a_245fad7c81c6.slice/crio-72c2befc366c070fd7eaf0223d95bbc8e170268d6a74a42c47ed4a7a8b2d3832 WatchSource:0}: Error finding container 72c2befc366c070fd7eaf0223d95bbc8e170268d6a74a42c47ed4a7a8b2d3832: Status 404 returned error can't find the container with id 72c2befc366c070fd7eaf0223d95bbc8e170268d6a74a42c47ed4a7a8b2d3832 Nov 02 03:40:27.705465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:27.705438 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" event={"ID":"51d9395d-42a0-49fd-a95a-245fad7c81c6","Type":"ContainerStarted","Data":"72c2befc366c070fd7eaf0223d95bbc8e170268d6a74a42c47ed4a7a8b2d3832"} Nov 02 03:40:28.712444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:28.712347 4868 generic.go:356] "Generic (PLEG): container finished" podID="51d9395d-42a0-49fd-a95a-245fad7c81c6" containerID="d8fbdd3b46ba5e6aa5674dc605aa8201df66fa57057d64b4c16247f3cb232286" exitCode=0 Nov 02 03:40:28.712444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:28.712436 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" event={"ID":"51d9395d-42a0-49fd-a95a-245fad7c81c6","Type":"ContainerDied","Data":"d8fbdd3b46ba5e6aa5674dc605aa8201df66fa57057d64b4c16247f3cb232286"} Nov 02 03:40:29.858411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.858384 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:29.897947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.897922 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9\" (UniqueName: \"kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9\") pod \"51d9395d-42a0-49fd-a95a-245fad7c81c6\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " Nov 02 03:40:29.898061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.898033 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca\") pod \"51d9395d-42a0-49fd-a95a-245fad7c81c6\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " Nov 02 03:40:29.898117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.898061 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca\") pod \"51d9395d-42a0-49fd-a95a-245fad7c81c6\" (UID: \"51d9395d-42a0-49fd-a95a-245fad7c81c6\") " Nov 02 03:40:29.898382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.898337 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca" pod "51d9395d-42a0-49fd-a95a-245fad7c81c6" (UID: "51d9395d-42a0-49fd-a95a-245fad7c81c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:40:29.899876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.899853 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9" pod "51d9395d-42a0-49fd-a95a-245fad7c81c6" (UID: "51d9395d-42a0-49fd-a95a-245fad7c81c6"). InnerVolumeSpecName "kube-api-access-528l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:40:29.899984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.899952 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca" pod "51d9395d-42a0-49fd-a95a-245fad7c81c6" (UID: "51d9395d-42a0-49fd-a95a-245fad7c81c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:40:29.999084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.999018 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:29.999084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.999046 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9395d-42a0-49fd-a95a-245fad7c81c6-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:29.999084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:29.999056 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528l9\" (UniqueName: \"kubernetes.io/projected/51d9395d-42a0-49fd-a95a-245fad7c81c6-kube-api-access-528l9\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:30.730337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:30.729989 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" event={"ID":"51d9395d-42a0-49fd-a95a-245fad7c81c6","Type":"ContainerDied","Data":"72c2befc366c070fd7eaf0223d95bbc8e170268d6a74a42c47ed4a7a8b2d3832"} Nov 02 03:40:30.730337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:30.730049 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72c2befc366c070fd7eaf0223d95bbc8e170268d6a74a42c47ed4a7a8b2d3832" Nov 02 03:40:30.730337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:30.730113 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b" Nov 02 03:40:31.249485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.247827 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5"] Nov 02 03:40:31.249485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.248933 4868 cpu_manager.go:414] "containerMap: removing container" podUID="51d9395d-42a0-49fd-a95a-245fad7c81c6" containerName="revoke-keys" Nov 02 03:40:31.249485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.248954 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d9395d-42a0-49fd-a95a-245fad7c81c6" containerName="revoke-keys" Nov 02 03:40:31.249485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.249279 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d9395d-42a0-49fd-a95a-245fad7c81c6" containerName="revoke-keys" Nov 02 03:40:31.256324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.256289 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.308692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.308662 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ad96f4-openshift-default-dockercfg-tbskh\"" Nov 02 03:40:31.312526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312498 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312534 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6qd\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312556 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312675 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312727 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312746 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312804 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312859 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.312883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.312881 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.361251 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.361211 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5"] Nov 02 03:40:31.445410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.438277 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.445410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.438330 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.445410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.438378 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.445410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.438877 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446168 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446320 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446226 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446320 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446293 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446331 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6qd\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446387 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446474 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.446910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.446871 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.447458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.447436 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.448067 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.448009 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.448783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.448735 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.451717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.451688 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.451907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.451884 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.503439 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.503376 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.503811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.503793 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6qd\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd\") pod \"e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.571089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.571043 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:31.998011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:31.997981 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5"] Nov 02 03:40:31.999462 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:40:31.999434 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9898e751_dc26_4f43_8979_fb1646684f98.slice/crio-81d726eec21f2224c45ee5aba9c2629d71478c78200253d44d97aec664bae96c WatchSource:0}: Error finding container 81d726eec21f2224c45ee5aba9c2629d71478c78200253d44d97aec664bae96c: Status 404 returned error can't find the container with id 81d726eec21f2224c45ee5aba9c2629d71478c78200253d44d97aec664bae96c Nov 02 03:40:32.001994 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:32.001964 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:40:32.002083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:32.002025 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:40:32.002083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:32.002054 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:40:32.744397 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:32.744342 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" event={"ID":"9898e751-dc26-4f43-8979-fb1646684f98","Type":"ContainerStarted","Data":"55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1"} Nov 02 03:40:32.744785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:32.744410 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" event={"ID":"9898e751-dc26-4f43-8979-fb1646684f98","Type":"ContainerStarted","Data":"81d726eec21f2224c45ee5aba9c2629d71478c78200253d44d97aec664bae96c"} Nov 02 03:40:32.962955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:32.962880 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podStartSLOduration=1.962861421 podStartE2EDuration="1.962861421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:40:32.961724546 +0000 UTC m=+3174.957425466" watchObservedRunningTime="2026-11-02 03:40:32.962861421 +0000 UTC m=+3174.958562428" Nov 02 03:40:33.445956 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:40:33.445925 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:40:33.571574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:33.571543 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:33.577161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:33.577143 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:33.753115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:33.753034 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:33.757466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:33.757433 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:40:42.890879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:42.890848 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4"] Nov 02 03:40:43.159066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.158976 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn"] Nov 02 03:40:43.166495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.166465 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.217205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.217172 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn"] Nov 02 03:40:43.218222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.218196 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:40:43.218389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.218318 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:40:43.314406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.314308 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.314858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.314827 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.315064 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.315046 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkc5\" (UniqueName: \"kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.416545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.416461 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.416699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.416550 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.416699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.416600 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkc5\" (UniqueName: \"kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.417097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.417073 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.418876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.418859 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.478327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.478017 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkc5\" (UniqueName: \"kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5\") pod \"maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.479782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.479756 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:43.860374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:43.860328 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn"] Nov 02 03:40:43.861461 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:40:43.861435 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026e2181_0211_4e91_9364_313f6843f7c4.slice/crio-26ce4257d1ad9772cfdda5452a528776e2a317abb438b0e79013a99def75d656 WatchSource:0}: Error finding container 26ce4257d1ad9772cfdda5452a528776e2a317abb438b0e79013a99def75d656: Status 404 returned error can't find the container with id 26ce4257d1ad9772cfdda5452a528776e2a317abb438b0e79013a99def75d656 Nov 02 03:40:44.818285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:44.818258 4868 generic.go:356] "Generic (PLEG): container finished" podID="026e2181-0211-4e91-9364-313f6843f7c4" containerID="5e3f2b6cc755e07cb0f9e6fa7fd175ad36d9ad3008904614696a3c4073300b9d" exitCode=0 Nov 02 03:40:44.818611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:44.818338 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" event={"ID":"026e2181-0211-4e91-9364-313f6843f7c4","Type":"ContainerDied","Data":"5e3f2b6cc755e07cb0f9e6fa7fd175ad36d9ad3008904614696a3c4073300b9d"} Nov 02 03:40:44.818611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:44.818407 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" event={"ID":"026e2181-0211-4e91-9364-313f6843f7c4","Type":"ContainerStarted","Data":"26ce4257d1ad9772cfdda5452a528776e2a317abb438b0e79013a99def75d656"} Nov 02 03:40:45.972707 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:45.972679 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:46.144753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.144724 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca\") pod \"026e2181-0211-4e91-9364-313f6843f7c4\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " Nov 02 03:40:46.144915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.144783 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5\" (UniqueName: \"kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5\") pod \"026e2181-0211-4e91-9364-313f6843f7c4\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " Nov 02 03:40:46.144915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.144828 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca\") pod \"026e2181-0211-4e91-9364-313f6843f7c4\" (UID: \"026e2181-0211-4e91-9364-313f6843f7c4\") " Nov 02 03:40:46.145134 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.145111 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca" pod "026e2181-0211-4e91-9364-313f6843f7c4" (UID: "026e2181-0211-4e91-9364-313f6843f7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:40:46.145239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.145218 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:46.146854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.146831 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca" pod "026e2181-0211-4e91-9364-313f6843f7c4" (UID: "026e2181-0211-4e91-9364-313f6843f7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:40:46.146927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.146903 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5" pod "026e2181-0211-4e91-9364-313f6843f7c4" (UID: "026e2181-0211-4e91-9364-313f6843f7c4"). InnerVolumeSpecName "kube-api-access-wtkc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:40:46.245868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.245843 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtkc5\" (UniqueName: \"kubernetes.io/projected/026e2181-0211-4e91-9364-313f6843f7c4-kube-api-access-wtkc5\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:46.245868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.245867 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026e2181-0211-4e91-9364-313f6843f7c4-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:46.832545 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.832506 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" Nov 02 03:40:46.832744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.832549 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn" event={"ID":"026e2181-0211-4e91-9364-313f6843f7c4","Type":"ContainerDied","Data":"26ce4257d1ad9772cfdda5452a528776e2a317abb438b0e79013a99def75d656"} Nov 02 03:40:46.832744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:46.832597 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26ce4257d1ad9772cfdda5452a528776e2a317abb438b0e79013a99def75d656" Nov 02 03:40:47.891936 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:47.891901 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" podUID="486e894c-3ba6-4c5d-9c65-09338df48932" containerName="payload-processing" containerID="cri-o://2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654" gracePeriod=25 Nov 02 03:40:48.153894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.153840 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:40:48.276868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.276840 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x\" (UniqueName: \"kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x\") pod \"486e894c-3ba6-4c5d-9c65-09338df48932\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " Nov 02 03:40:48.277344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.277292 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume\") pod \"486e894c-3ba6-4c5d-9c65-09338df48932\" (UID: \"486e894c-3ba6-4c5d-9c65-09338df48932\") " Nov 02 03:40:48.278010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.277981 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume" pod "486e894c-3ba6-4c5d-9c65-09338df48932" (UID: "486e894c-3ba6-4c5d-9c65-09338df48932"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:40:48.279362 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.279316 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x" pod "486e894c-3ba6-4c5d-9c65-09338df48932" (UID: "486e894c-3ba6-4c5d-9c65-09338df48932"). InnerVolumeSpecName "kube-api-access-rsh5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:40:48.378856 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.378828 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486e894c-3ba6-4c5d-9c65-09338df48932-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:48.378856 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.378852 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsh5x\" (UniqueName: \"kubernetes.io/projected/486e894c-3ba6-4c5d-9c65-09338df48932-kube-api-access-rsh5x\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:40:48.844709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.844629 4868 generic.go:356] "Generic (PLEG): container finished" podID="486e894c-3ba6-4c5d-9c65-09338df48932" containerID="2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654" exitCode=0 Nov 02 03:40:48.844709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.844690 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" Nov 02 03:40:48.844905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.844695 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" event={"ID":"486e894c-3ba6-4c5d-9c65-09338df48932","Type":"ContainerDied","Data":"2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654"} Nov 02 03:40:48.844905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.844827 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4" event={"ID":"486e894c-3ba6-4c5d-9c65-09338df48932","Type":"ContainerDied","Data":"e1a3753bc8e591edef1d579576ea62b5de8bdd827065efb7d33ba65c986d273b"} Nov 02 03:40:48.844905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.844849 4868 scope.go:122] "RemoveContainer" containerID="2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654" Nov 02 03:40:48.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.856581 4868 scope.go:122] "RemoveContainer" containerID="2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654" Nov 02 03:40:48.857674 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:40:48.857647 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654\": container with ID starting with 2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654 not found: ID does not exist" containerID="2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654" Nov 02 03:40:48.857781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:48.857685 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654"} err="failed to get container status \"2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654\": rpc error: code = NotFound desc = could not find container \"2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654\": container with ID starting with 2254fbe1bf10ab491c822bd04828d8dc2602bc852882ee53efd8260bb1fe4654 not found: ID does not exist" Nov 02 03:40:49.006727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:49.006702 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4"] Nov 02 03:40:49.055816 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:49.055784 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e44998-55d9d7867c-9sqz4"] Nov 02 03:40:50.113921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:40:50.113886 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486e894c-3ba6-4c5d-9c65-09338df48932" path="/var/lib/kubelet/pods/486e894c-3ba6-4c5d-9c65-09338df48932/volumes" Nov 02 03:41:03.445910 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:41:03.445876 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:41:07.332103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.332060 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh"] Nov 02 03:41:07.333442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.333413 4868 cpu_manager.go:414] "containerMap: removing container" podUID="486e894c-3ba6-4c5d-9c65-09338df48932" containerName="payload-processing" Nov 02 03:41:07.333620 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.333605 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486e894c-3ba6-4c5d-9c65-09338df48932" containerName="payload-processing" Nov 02 03:41:07.333772 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.333758 4868 cpu_manager.go:414] "containerMap: removing container" podUID="026e2181-0211-4e91-9364-313f6843f7c4" containerName="revoke-keys" Nov 02 03:41:07.333861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.333850 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026e2181-0211-4e91-9364-313f6843f7c4" containerName="revoke-keys" Nov 02 03:41:07.334091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.334077 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="486e894c-3ba6-4c5d-9c65-09338df48932" containerName="payload-processing" Nov 02 03:41:07.334196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.334186 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="026e2181-0211-4e91-9364-313f6843f7c4" containerName="revoke-keys" Nov 02 03:41:07.348373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.341497 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.415926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.415887 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.416266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.416248 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.416478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.416458 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.416691 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.416673 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.416872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.416857 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknks\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.417018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.417003 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.417257 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.417236 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.417555 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.417534 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.417756 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.417741 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.441342 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.441297 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh"] Nov 02 03:41:07.460536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.460421 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f68c7e-openshift-default-dockercfg-9xx98\"" Nov 02 03:41:07.519128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.519091 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.519399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.519377 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.519556 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.519541 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.519709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.519696 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.519827 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.519814 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.519946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.519934 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.520054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.520043 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.520171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.520158 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknks\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.520268 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.520256 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.520824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.520798 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.520980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.520856 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.521326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.521307 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.521955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.521925 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.522506 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.522480 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.532391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.526090 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.532391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.527937 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.578709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.578677 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.578840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.578777 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknks\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks\") pod \"e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:07.660998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:07.660969 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:08.097616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:08.097532 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh"] Nov 02 03:41:08.142815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:08.142779 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:41:08.142893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:08.142869 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:41:08.142940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:08.142914 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:41:08.971087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:08.971050 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" event={"ID":"23b7191e-00d3-4cae-8883-2ca020f5467b","Type":"ContainerStarted","Data":"730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae"} Nov 02 03:41:08.971559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:08.971123 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" event={"ID":"23b7191e-00d3-4cae-8883-2ca020f5467b","Type":"ContainerStarted","Data":"7da1dae53d401ee616e3df76ed36f609c6986a01c15dd0b1b3a8a715ff38d609"} Nov 02 03:41:09.087864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:09.087818 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podStartSLOduration=2.087805753 podStartE2EDuration="2.087805753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:41:09.085181402 +0000 UTC m=+3211.080882349" watchObservedRunningTime="2026-11-02 03:41:09.087805753 +0000 UTC m=+3211.083506696" Nov 02 03:41:09.661780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:09.661753 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:09.667403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:09.667383 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:09.977743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:09.977650 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:09.978642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:09.978614 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:41:15.584372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.583969 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs"] Nov 02 03:41:15.608376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.598257 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.642269 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.642236 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs"] Nov 02 03:41:15.687643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.687592 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f68c7e-dockercfg-gn297\"" Nov 02 03:41:15.687643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.687597 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f68c7e\"" Nov 02 03:41:15.731985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.731945 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9w6\" (UniqueName: \"kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6\") pod \"payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.732191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.732170 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.832945 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.832912 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.833087 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.832971 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9w6\" (UniqueName: \"kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6\") pod \"payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.833637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.833617 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.891749 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.891724 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9w6\" (UniqueName: \"kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6\") pod \"payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:15.933882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:15.933860 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:16.318656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:16.318634 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs"] Nov 02 03:41:16.320112 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:41:16.320086 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528bbd62_8941_4d37_84f8_e8e8618bc8bd.slice/crio-a828dcc29a10d9ebbbdec4438d0dbbd0905484506c278c8287d978f0b92bd2a2 WatchSource:0}: Error finding container a828dcc29a10d9ebbbdec4438d0dbbd0905484506c278c8287d978f0b92bd2a2: Status 404 returned error can't find the container with id a828dcc29a10d9ebbbdec4438d0dbbd0905484506c278c8287d978f0b92bd2a2 Nov 02 03:41:17.027399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.027334 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" event={"ID":"528bbd62-8941-4d37-84f8-e8e8618bc8bd","Type":"ContainerStarted","Data":"f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63"} Nov 02 03:41:17.027860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.027820 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" event={"ID":"528bbd62-8941-4d37-84f8-e8e8618bc8bd","Type":"ContainerStarted","Data":"a828dcc29a10d9ebbbdec4438d0dbbd0905484506c278c8287d978f0b92bd2a2"} Nov 02 03:41:17.036171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.036144 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:17.149035 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.148971 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" podStartSLOduration=2.148956925 podStartE2EDuration="2.148956925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:41:17.148252054 +0000 UTC m=+3219.143952999" watchObservedRunningTime="2026-11-02 03:41:17.148956925 +0000 UTC m=+3219.144657869" Nov 02 03:41:17.380128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.380095 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm"] Nov 02 03:41:17.380324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.380304 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" podUID="9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" containerName="maas-api" containerID="cri-o://d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c" gracePeriod=30 Nov 02 03:41:17.730828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.730796 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:41:17.865449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.865410 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\") pod \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " Nov 02 03:41:17.865633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.865504 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs\") pod \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " Nov 02 03:41:17.865633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.865551 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca\") pod \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " Nov 02 03:41:17.865633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.865592 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx\" (UniqueName: \"kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx\") pod \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\" (UID: \"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb\") " Nov 02 03:41:17.866576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.866546 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca" pod "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" (UID: "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:41:17.869125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.869068 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs" pod "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" (UID: "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:41:17.871509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.871481 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx" pod "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" (UID: "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb"). InnerVolumeSpecName "kube-api-access-qsnnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:17.874653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.874617 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls" pod "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" (UID: "9954f9a7-7914-4d15-b9f3-2dd428b3c2cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:41:17.967752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.967688 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:17.967752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.967714 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:17.967752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.967724 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnnx\" (UniqueName: \"kubernetes.io/projected/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-kube-api-access-qsnnx\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:17.967752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:17.967734 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:18.043085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.039537 4868 generic.go:356] "Generic (PLEG): container finished" podID="9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" containerID="d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c" exitCode=0 Nov 02 03:41:18.043085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.040665 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" Nov 02 03:41:18.044547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.044218 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" event={"ID":"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb","Type":"ContainerDied","Data":"d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c"} Nov 02 03:41:18.044547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.044277 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm" event={"ID":"9954f9a7-7914-4d15-b9f3-2dd428b3c2cb","Type":"ContainerDied","Data":"fd6be8ce1c5cd193ad7a049e6f21f96278c1937e0e0abe39e72f90b1020a72d5"} Nov 02 03:41:18.044547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.044308 4868 scope.go:122] "RemoveContainer" containerID="d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c" Nov 02 03:41:18.057179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.057159 4868 scope.go:122] "RemoveContainer" containerID="d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c" Nov 02 03:41:18.057484 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:41:18.057463 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c\": container with ID starting with d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c not found: ID does not exist" containerID="d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c" Nov 02 03:41:18.057537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.057492 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c"} err="failed to get container status \"d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c\": rpc error: code = NotFound desc = could not find container \"d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c\": container with ID starting with d59f145b28e611d7319e9064f47c91faf9de354df143e6f824b04b0547a89c4c not found: ID does not exist" Nov 02 03:41:18.116750 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.116724 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8"] Nov 02 03:41:18.116945 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.116925 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" containerID="cri-o://680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69" gracePeriod=30 Nov 02 03:41:18.206175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.206145 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm"] Nov 02 03:41:18.209735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:18.209708 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e44998-84979bf95b-782jm"] Nov 02 03:41:19.798811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:19.798775 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b"] Nov 02 03:41:19.847904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:19.847875 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-e44998-5rk6b"] Nov 02 03:41:20.115345 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:20.115308 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d9395d-42a0-49fd-a95a-245fad7c81c6" path="/var/lib/kubelet/pods/51d9395d-42a0-49fd-a95a-245fad7c81c6/volumes" Nov 02 03:41:20.115905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:20.115888 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" path="/var/lib/kubelet/pods/9954f9a7-7914-4d15-b9f3-2dd428b3c2cb/volumes" Nov 02 03:41:22.315595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:22.315567 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dmsh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:41:22.315976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:22.315625 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:41:23.374040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.374014 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:41:23.522264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522197 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522419 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522293 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522419 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522342 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522457 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522485 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:23.522505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522499 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522532 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522572 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522564 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:23.522626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522609 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522625 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:23.522832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522639 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo\") pod \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\" (UID: \"2ea2374b-e26e-41e5-b53f-f57fe695ecad\") " Nov 02 03:41:23.522947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522823 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:23.522994 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.522952 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:41:23.523095 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.523073 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.523192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.523098 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.523192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.523111 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.523192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.523123 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.523192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.523135 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.524792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.524763 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:41:23.524871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.524775 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "kube-api-access-8d24q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:23.525041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.525022 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:23.525096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.525056 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token" pod "2ea2374b-e26e-41e5-b53f-f57fe695ecad" (UID: "2ea2374b-e26e-41e5-b53f-f57fe695ecad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:23.624662 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.624634 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d24q\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-kube-api-access-8d24q\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.624662 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.624661 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.624841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.624671 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:23.624841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:23.624680 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea2374b-e26e-41e5-b53f-f57fe695ecad-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:24.078299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.078259 4868 generic.go:356] "Generic (PLEG): container finished" podID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerID="680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69" exitCode=0 Nov 02 03:41:24.078484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.078307 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" Nov 02 03:41:24.078484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.078342 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" event={"ID":"2ea2374b-e26e-41e5-b53f-f57fe695ecad","Type":"ContainerDied","Data":"680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69"} Nov 02 03:41:24.078484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.078427 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8" event={"ID":"2ea2374b-e26e-41e5-b53f-f57fe695ecad","Type":"ContainerDied","Data":"dc15fc02b372cddba040837b4dfd6ca7a42eaf222db93cc973aa711743942bdd"} Nov 02 03:41:24.078484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.078457 4868 scope.go:122] "RemoveContainer" containerID="680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69" Nov 02 03:41:24.089490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.089469 4868 scope.go:122] "RemoveContainer" containerID="680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69" Nov 02 03:41:24.089771 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:41:24.089744 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69\": container with ID starting with 680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69 not found: ID does not exist" containerID="680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69" Nov 02 03:41:24.089837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.089784 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69"} err="failed to get container status \"680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69\": rpc error: code = NotFound desc = could not find container \"680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69\": container with ID starting with 680291dd7bcb572f38543e6f34598afd6676372dec066f5ad9c119362b5bcf69 not found: ID does not exist" Nov 02 03:41:24.251898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.251862 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8"] Nov 02 03:41:24.300892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:24.300861 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dmsh8"] Nov 02 03:41:25.971684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:25.971644 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn"] Nov 02 03:41:26.021011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:26.020982 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0a04e0-wkwvn"] Nov 02 03:41:26.135076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:26.135039 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026e2181-0211-4e91-9364-313f6843f7c4" path="/var/lib/kubelet/pods/026e2181-0211-4e91-9364-313f6843f7c4/volumes" Nov 02 03:41:26.135593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:26.135577 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" path="/var/lib/kubelet/pods/2ea2374b-e26e-41e5-b53f-f57fe695ecad/volumes" Nov 02 03:41:27.233527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.233493 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl"] Nov 02 03:41:27.234009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.233975 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" containerName="maas-api" Nov 02 03:41:27.234009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.233988 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" containerName="maas-api" Nov 02 03:41:27.234126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.234017 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" Nov 02 03:41:27.234126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.234023 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" Nov 02 03:41:27.234230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.234126 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9954f9a7-7914-4d15-b9f3-2dd428b3c2cb" containerName="maas-api" Nov 02 03:41:27.234230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.234141 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea2374b-e26e-41e5-b53f-f57fe695ecad" containerName="istio-proxy" Nov 02 03:41:27.238900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.238878 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.294046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.294014 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl"] Nov 02 03:41:27.297476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.297446 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.297595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.297575 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xl6\" (UniqueName: \"kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.297646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.297625 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.340150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.340126 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:41:27.340264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.340168 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:41:27.398189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.398160 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.398306 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.398239 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xl6\" (UniqueName: \"kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.398306 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.398278 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.398801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.398779 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.400747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.400724 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.454376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.454331 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xl6\" (UniqueName: \"kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.552688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.552628 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:27.924646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:27.924623 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl"] Nov 02 03:41:27.925873 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:41:27.925840 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01c6772_1a9d_4af3_9fb8_6f607c86a94d.slice/crio-e545ce0a45111bf8b5608052ac44bc2b4bd9fa3dd76ee835d04fb8da80beaea5 WatchSource:0}: Error finding container e545ce0a45111bf8b5608052ac44bc2b4bd9fa3dd76ee835d04fb8da80beaea5: Status 404 returned error can't find the container with id e545ce0a45111bf8b5608052ac44bc2b4bd9fa3dd76ee835d04fb8da80beaea5 Nov 02 03:41:28.043324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:28.043305 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:41:28.113886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:28.113850 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" event={"ID":"a01c6772-1a9d-4af3-9fb8-6f607c86a94d","Type":"ContainerStarted","Data":"e545ce0a45111bf8b5608052ac44bc2b4bd9fa3dd76ee835d04fb8da80beaea5"} Nov 02 03:41:29.124936 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:29.124904 4868 generic.go:356] "Generic (PLEG): container finished" podID="a01c6772-1a9d-4af3-9fb8-6f607c86a94d" containerID="692806991a25e23719379dd2807cb0cf35e240bb5358c3fba4cf36d0fef2aacb" exitCode=0 Nov 02 03:41:29.125659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:29.125629 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" event={"ID":"a01c6772-1a9d-4af3-9fb8-6f607c86a94d","Type":"ContainerDied","Data":"692806991a25e23719379dd2807cb0cf35e240bb5358c3fba4cf36d0fef2aacb"} Nov 02 03:41:30.274285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.274261 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:30.436974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.436901 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca\") pod \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " Nov 02 03:41:30.437123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.437015 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6\" (UniqueName: \"kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6\") pod \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " Nov 02 03:41:30.437123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.437061 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca\") pod \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\" (UID: \"a01c6772-1a9d-4af3-9fb8-6f607c86a94d\") " Nov 02 03:41:30.437318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.437284 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca" pod "a01c6772-1a9d-4af3-9fb8-6f607c86a94d" (UID: "a01c6772-1a9d-4af3-9fb8-6f607c86a94d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:41:30.437585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.437564 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:30.439177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.439144 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6" pod "a01c6772-1a9d-4af3-9fb8-6f607c86a94d" (UID: "a01c6772-1a9d-4af3-9fb8-6f607c86a94d"). InnerVolumeSpecName "kube-api-access-w7xl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:30.439291 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.439192 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca" pod "a01c6772-1a9d-4af3-9fb8-6f607c86a94d" (UID: "a01c6772-1a9d-4af3-9fb8-6f607c86a94d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:41:30.538465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.538437 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xl6\" (UniqueName: \"kubernetes.io/projected/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-kube-api-access-w7xl6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:30.538465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:30.538463 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a01c6772-1a9d-4af3-9fb8-6f607c86a94d-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:31.137595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:31.137566 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" Nov 02 03:41:31.137755 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:31.137601 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl" event={"ID":"a01c6772-1a9d-4af3-9fb8-6f607c86a94d","Type":"ContainerDied","Data":"e545ce0a45111bf8b5608052ac44bc2b4bd9fa3dd76ee835d04fb8da80beaea5"} Nov 02 03:41:31.137755 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:31.137649 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e545ce0a45111bf8b5608052ac44bc2b4bd9fa3dd76ee835d04fb8da80beaea5" Nov 02 03:41:32.515901 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:32.515870 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt"] Nov 02 03:41:32.516421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:32.516134 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" containerID="cri-o://38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250" gracePeriod=30 Nov 02 03:41:33.333734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.333694 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9"] Nov 02 03:41:33.334258 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.334238 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a01c6772-1a9d-4af3-9fb8-6f607c86a94d" containerName="revoke-keys" Nov 02 03:41:33.334391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.334258 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01c6772-1a9d-4af3-9fb8-6f607c86a94d" containerName="revoke-keys" Nov 02 03:41:33.334681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.334436 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01c6772-1a9d-4af3-9fb8-6f607c86a94d" containerName="revoke-keys" Nov 02 03:41:33.337906 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.337887 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.390480 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.390450 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:41:33.390720 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.390570 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:41:33.393943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.393916 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9"] Nov 02 03:41:33.446136 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:41:33.446099 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:41:33.470498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.470471 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526kl\" (UniqueName: \"kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.470630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.470503 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.470674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.470632 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.571485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.571460 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526kl\" (UniqueName: \"kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.571819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.571488 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.571819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.571550 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.572123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.572103 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.573772 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.573743 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.627555 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.627527 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526kl\" (UniqueName: \"kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl\") pod \"maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:33.660651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:33.660630 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:34.035315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:34.035287 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9"] Nov 02 03:41:34.035954 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:41:34.035930 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc156cf3b_b612_4099_8eab_d0bf071a9733.slice/crio-e498d51373bfde5e8e10ba3215d6088993b90d95ab514caa61f88aeb53c1e604 WatchSource:0}: Error finding container e498d51373bfde5e8e10ba3215d6088993b90d95ab514caa61f88aeb53c1e604: Status 404 returned error can't find the container with id e498d51373bfde5e8e10ba3215d6088993b90d95ab514caa61f88aeb53c1e604 Nov 02 03:41:34.155484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:34.155456 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" event={"ID":"c156cf3b-b612-4099-8eab-d0bf071a9733","Type":"ContainerStarted","Data":"e498d51373bfde5e8e10ba3215d6088993b90d95ab514caa61f88aeb53c1e604"} Nov 02 03:41:34.747699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:34.747664 4868 patch_prober.go:29] interesting pod/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:41:34.748136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:34.747735 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:41:35.162828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:35.162797 4868 generic.go:356] "Generic (PLEG): container finished" podID="c156cf3b-b612-4099-8eab-d0bf071a9733" containerID="9e699ab612ae60c72afa78cde0670db4db97f59b54305fdaf2b6ba655192ec04" exitCode=0 Nov 02 03:41:35.162984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:35.162873 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" event={"ID":"c156cf3b-b612-4099-8eab-d0bf071a9733","Type":"ContainerDied","Data":"9e699ab612ae60c72afa78cde0670db4db97f59b54305fdaf2b6ba655192ec04"} Nov 02 03:41:36.331386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.331344 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:36.503845 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.503777 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl\" (UniqueName: \"kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl\") pod \"c156cf3b-b612-4099-8eab-d0bf071a9733\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " Nov 02 03:41:36.503989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.503861 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca\") pod \"c156cf3b-b612-4099-8eab-d0bf071a9733\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " Nov 02 03:41:36.504042 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.504004 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca\") pod \"c156cf3b-b612-4099-8eab-d0bf071a9733\" (UID: \"c156cf3b-b612-4099-8eab-d0bf071a9733\") " Nov 02 03:41:36.504427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.504401 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca" pod "c156cf3b-b612-4099-8eab-d0bf071a9733" (UID: "c156cf3b-b612-4099-8eab-d0bf071a9733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:41:36.505907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.505878 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl" pod "c156cf3b-b612-4099-8eab-d0bf071a9733" (UID: "c156cf3b-b612-4099-8eab-d0bf071a9733"). InnerVolumeSpecName "kube-api-access-526kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:36.506007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.505945 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca" pod "c156cf3b-b612-4099-8eab-d0bf071a9733" (UID: "c156cf3b-b612-4099-8eab-d0bf071a9733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:41:36.605428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.605405 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:36.605428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.605429 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-526kl\" (UniqueName: \"kubernetes.io/projected/c156cf3b-b612-4099-8eab-d0bf071a9733-kube-api-access-526kl\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:36.605563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:36.605438 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c156cf3b-b612-4099-8eab-d0bf071a9733-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:37.176152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.176120 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" Nov 02 03:41:37.176318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.176170 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9" event={"ID":"c156cf3b-b612-4099-8eab-d0bf071a9733","Type":"ContainerDied","Data":"e498d51373bfde5e8e10ba3215d6088993b90d95ab514caa61f88aeb53c1e604"} Nov 02 03:41:37.176318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.176225 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e498d51373bfde5e8e10ba3215d6088993b90d95ab514caa61f88aeb53c1e604" Nov 02 03:41:37.787469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.787408 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:41:37.917232 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917206 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917243 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917271 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917395 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917539 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917498 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917543 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:37.917649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917590 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917614 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:41:37.917649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917640 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917683 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917721 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs\") pod \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\" (UID: \"e35fd55b-dbd5-4fa0-9782-fd36825c9850\") " Nov 02 03:41:37.917927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.917892 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:37.918037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918003 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:37.918136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918107 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:37.918229 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918208 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:37.918292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918235 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:37.918292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918251 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:37.918292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918266 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:37.918292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.918280 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:37.919525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.919496 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "kube-api-access-7qm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:37.919643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.919629 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:41:37.919726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.919707 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:41:37.919778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:37.919766 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy" pod "e35fd55b-dbd5-4fa0-9782-fd36825c9850" (UID: "e35fd55b-dbd5-4fa0-9782-fd36825c9850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:41:38.018973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.018945 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:38.018973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.018970 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:38.019146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.018980 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qm8g\" (UniqueName: \"kubernetes.io/projected/e35fd55b-dbd5-4fa0-9782-fd36825c9850-kube-api-access-7qm8g\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:38.019146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.018990 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35fd55b-dbd5-4fa0-9782-fd36825c9850-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:41:38.187215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.187177 4868 generic.go:356] "Generic (PLEG): container finished" podID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerID="38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250" exitCode=0 Nov 02 03:41:38.188103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.187373 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" event={"ID":"e35fd55b-dbd5-4fa0-9782-fd36825c9850","Type":"ContainerDied","Data":"38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250"} Nov 02 03:41:38.188103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.187466 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" event={"ID":"e35fd55b-dbd5-4fa0-9782-fd36825c9850","Type":"ContainerDied","Data":"7eee050d2ffd99391d27b13f88c888f6b1a0cda023d56abbba0c7a30b69fb6e0"} Nov 02 03:41:38.188103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.187504 4868 scope.go:122] "RemoveContainer" containerID="38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250" Nov 02 03:41:38.188103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.187705 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt" Nov 02 03:41:38.200332 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.200307 4868 scope.go:122] "RemoveContainer" containerID="4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461" Nov 02 03:41:38.209564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.209543 4868 scope.go:122] "RemoveContainer" containerID="38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250" Nov 02 03:41:38.209824 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:41:38.209802 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250\": container with ID starting with 38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250 not found: ID does not exist" containerID="38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250" Nov 02 03:41:38.209907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.209830 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250"} err="failed to get container status \"38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250\": rpc error: code = NotFound desc = could not find container \"38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250\": container with ID starting with 38e81b488755094d5c5f19c03483d0f0277503bc9ee1c82db71fbaaa9c8fb250 not found: ID does not exist" Nov 02 03:41:38.209907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.209848 4868 scope.go:122] "RemoveContainer" containerID="4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461" Nov 02 03:41:38.210091 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:41:38.210070 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461\": container with ID starting with 4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461 not found: ID does not exist" containerID="4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461" Nov 02 03:41:38.210188 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.210096 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461"} err="failed to get container status \"4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461\": rpc error: code = NotFound desc = could not find container \"4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461\": container with ID starting with 4bb7fad91b490c66a3762cf8c56e1c612f2b25665c94fc2317cd383202a74461 not found: ID does not exist" Nov 02 03:41:38.351980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.351947 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt"] Nov 02 03:41:38.356754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:38.356727 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0a04e0-openshift-default-76ff7fc957-khvzt"] Nov 02 03:41:40.114639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:40.114605 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" path="/var/lib/kubelet/pods/e35fd55b-dbd5-4fa0-9782-fd36825c9850/volumes" Nov 02 03:41:51.996666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:51.996631 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9"] Nov 02 03:41:52.045695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:52.045665 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cb56e9-6cgl9"] Nov 02 03:41:52.114694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:41:52.114665 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c156cf3b-b612-4099-8eab-d0bf071a9733" path="/var/lib/kubelet/pods/c156cf3b-b612-4099-8eab-d0bf071a9733/volumes" Nov 02 03:42:03.466165 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:42:03.466122 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:42:06.398137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398098 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr"] Nov 02 03:42:06.398624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398558 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" Nov 02 03:42:06.398624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398570 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" Nov 02 03:42:06.398624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398616 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c156cf3b-b612-4099-8eab-d0bf071a9733" containerName="revoke-keys" Nov 02 03:42:06.398624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398621 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c156cf3b-b612-4099-8eab-d0bf071a9733" containerName="revoke-keys" Nov 02 03:42:06.398760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398699 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c156cf3b-b612-4099-8eab-d0bf071a9733" containerName="revoke-keys" Nov 02 03:42:06.398760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398708 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" Nov 02 03:42:06.398760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.398724 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" Nov 02 03:42:06.403330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.403311 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.460135 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.460103 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr"] Nov 02 03:42:06.478701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.478664 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xt4\" (UniqueName: \"kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.478873 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.478847 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.478970 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.478955 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.500225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.500201 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:42:06.506085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.506063 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:42:06.581759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.581699 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xt4\" (UniqueName: \"kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.582164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.582133 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.582302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.582236 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.583029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.583006 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.584711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.584689 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.649090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.649024 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xt4\" (UniqueName: \"kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4\") pod \"maas-api-revoke-sub-407a3d33-xjztr\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:06.716684 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:06.716657 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:07.118100 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:07.118071 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr"] Nov 02 03:42:07.119916 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:42:07.119882 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66eab69_0a4c_4779_89d3_8b2b20716b36.slice/crio-44544ec779e1df388447bfa4a17d11b713ba353dfab506dc3031d5947c785bf7 WatchSource:0}: Error finding container 44544ec779e1df388447bfa4a17d11b713ba353dfab506dc3031d5947c785bf7: Status 404 returned error can't find the container with id 44544ec779e1df388447bfa4a17d11b713ba353dfab506dc3031d5947c785bf7 Nov 02 03:42:07.384098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:07.384063 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" event={"ID":"f66eab69-0a4c-4779-89d3-8b2b20716b36","Type":"ContainerStarted","Data":"44544ec779e1df388447bfa4a17d11b713ba353dfab506dc3031d5947c785bf7"} Nov 02 03:42:08.391654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:08.391614 4868 generic.go:356] "Generic (PLEG): container finished" podID="f66eab69-0a4c-4779-89d3-8b2b20716b36" containerID="118bfada98a6dfea68149c123fba270a8ebd8a5dc96ec6097af5c023acbfc6df" exitCode=0 Nov 02 03:42:08.392005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:08.391676 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" event={"ID":"f66eab69-0a4c-4779-89d3-8b2b20716b36","Type":"ContainerDied","Data":"118bfada98a6dfea68149c123fba270a8ebd8a5dc96ec6097af5c023acbfc6df"} Nov 02 03:42:09.540533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.540502 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:09.715979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.715898 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca\") pod \"f66eab69-0a4c-4779-89d3-8b2b20716b36\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " Nov 02 03:42:09.716122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.716036 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4\" (UniqueName: \"kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4\") pod \"f66eab69-0a4c-4779-89d3-8b2b20716b36\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " Nov 02 03:42:09.716175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.716128 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca\") pod \"f66eab69-0a4c-4779-89d3-8b2b20716b36\" (UID: \"f66eab69-0a4c-4779-89d3-8b2b20716b36\") " Nov 02 03:42:09.716263 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.716237 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca" pod "f66eab69-0a4c-4779-89d3-8b2b20716b36" (UID: "f66eab69-0a4c-4779-89d3-8b2b20716b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:42:09.716651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.716627 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:09.718151 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.718124 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca" pod "f66eab69-0a4c-4779-89d3-8b2b20716b36" (UID: "f66eab69-0a4c-4779-89d3-8b2b20716b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:42:09.718233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.718190 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4" pod "f66eab69-0a4c-4779-89d3-8b2b20716b36" (UID: "f66eab69-0a4c-4779-89d3-8b2b20716b36"). InnerVolumeSpecName "kube-api-access-f6xt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:42:09.817705 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.817677 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66eab69-0a4c-4779-89d3-8b2b20716b36-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:09.817818 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:09.817710 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xt4\" (UniqueName: \"kubernetes.io/projected/f66eab69-0a4c-4779-89d3-8b2b20716b36-kube-api-access-f6xt4\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:10.403661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:10.403592 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" Nov 02 03:42:10.403797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:10.403592 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr" event={"ID":"f66eab69-0a4c-4779-89d3-8b2b20716b36","Type":"ContainerDied","Data":"44544ec779e1df388447bfa4a17d11b713ba353dfab506dc3031d5947c785bf7"} Nov 02 03:42:10.403797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:10.403712 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44544ec779e1df388447bfa4a17d11b713ba353dfab506dc3031d5947c785bf7" Nov 02 03:42:11.410100 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:11.410029 4868 generic.go:356] "Generic (PLEG): container finished" podID="9898e751-dc26-4f43-8979-fb1646684f98" containerID="55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1" exitCode=0 Nov 02 03:42:11.410609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:11.410106 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" event={"ID":"9898e751-dc26-4f43-8979-fb1646684f98","Type":"ContainerDied","Data":"55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1"} Nov 02 03:42:11.410609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:11.410488 4868 scope.go:122] "RemoveContainer" containerID="55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1" Nov 02 03:42:11.411879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:11.411854 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:42:11.411949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:11.411918 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:42:11.411987 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:11.411953 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:42:12.417921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:12.417880 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" event={"ID":"9898e751-dc26-4f43-8979-fb1646684f98","Type":"ContainerStarted","Data":"3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e"} Nov 02 03:42:12.572055 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:12.572021 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:42:12.572745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:12.572719 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 03:42:12.572796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:12.572772 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 03:42:13.571818 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:13.571783 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 03:42:13.572219 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:13.571845 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 03:42:14.572164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:14.572131 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 03:42:14.572605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:14.572192 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 03:42:15.572004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:15.571968 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 03:42:15.572187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:15.572031 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 03:42:16.571563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:16.571520 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 03:42:16.571734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:16.571579 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 03:42:17.576737 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:17.576707 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:42:17.577230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:17.577213 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:42:17.577967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:17.577946 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:42:24.449202 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.449107 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg"] Nov 02 03:42:24.449656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.449600 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" Nov 02 03:42:24.449656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.449613 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e35fd55b-dbd5-4fa0-9782-fd36825c9850" containerName="istio-proxy" Nov 02 03:42:24.449656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.449656 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f66eab69-0a4c-4779-89d3-8b2b20716b36" containerName="revoke-keys" Nov 02 03:42:24.449774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.449661 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66eab69-0a4c-4779-89d3-8b2b20716b36" containerName="revoke-keys" Nov 02 03:42:24.449774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.449759 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66eab69-0a4c-4779-89d3-8b2b20716b36" containerName="revoke-keys" Nov 02 03:42:24.453000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.452983 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.506637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.506610 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:42:24.510263 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.510238 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg"] Nov 02 03:42:24.555557 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.555528 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:42:24.589708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.589679 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7wz\" (UniqueName: \"kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.589830 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.589790 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.589880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.589826 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.690726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.690701 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.690853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.690742 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.690923 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.690850 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7wz\" (UniqueName: \"kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.691500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.691477 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.693385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.693366 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.746792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.746745 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7wz\" (UniqueName: \"kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz\") pod \"maas-api-revoke-sub-f0588b02-4rcdg\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:24.765495 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:24.765477 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:25.135339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:25.135313 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg"] Nov 02 03:42:25.138072 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:42:25.138045 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465fd062_d546_453a_b343_95b93ae60c0d.slice/crio-56bf43c0e44c0e81034cd0bc01ea8023a85bd78c23732efa7f47876231c1dda7 WatchSource:0}: Error finding container 56bf43c0e44c0e81034cd0bc01ea8023a85bd78c23732efa7f47876231c1dda7: Status 404 returned error can't find the container with id 56bf43c0e44c0e81034cd0bc01ea8023a85bd78c23732efa7f47876231c1dda7 Nov 02 03:42:25.514556 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:25.514524 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" event={"ID":"465fd062-d546-453a-b343-95b93ae60c0d","Type":"ContainerStarted","Data":"56bf43c0e44c0e81034cd0bc01ea8023a85bd78c23732efa7f47876231c1dda7"} Nov 02 03:42:26.522309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:26.522278 4868 generic.go:356] "Generic (PLEG): container finished" podID="465fd062-d546-453a-b343-95b93ae60c0d" containerID="526410c6af912dd685d8b4b9a90ee42675ac8bca1335469d5c2a2b92cc214c2a" exitCode=0 Nov 02 03:42:26.522743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:26.522374 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" event={"ID":"465fd062-d546-453a-b343-95b93ae60c0d","Type":"ContainerDied","Data":"526410c6af912dd685d8b4b9a90ee42675ac8bca1335469d5c2a2b92cc214c2a"} Nov 02 03:42:27.686703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.686672 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:27.828619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.828519 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz\" (UniqueName: \"kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz\") pod \"465fd062-d546-453a-b343-95b93ae60c0d\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " Nov 02 03:42:27.828619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.828583 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca\") pod \"465fd062-d546-453a-b343-95b93ae60c0d\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " Nov 02 03:42:27.828842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.828758 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca\") pod \"465fd062-d546-453a-b343-95b93ae60c0d\" (UID: \"465fd062-d546-453a-b343-95b93ae60c0d\") " Nov 02 03:42:27.829326 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.829294 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca" pod "465fd062-d546-453a-b343-95b93ae60c0d" (UID: "465fd062-d546-453a-b343-95b93ae60c0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:42:27.831723 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.831411 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca" pod "465fd062-d546-453a-b343-95b93ae60c0d" (UID: "465fd062-d546-453a-b343-95b93ae60c0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:42:27.832025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.831985 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz" pod "465fd062-d546-453a-b343-95b93ae60c0d" (UID: "465fd062-d546-453a-b343-95b93ae60c0d"). InnerVolumeSpecName "kube-api-access-jw7wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:42:27.852016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.851989 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q"] Nov 02 03:42:27.852940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.852918 4868 cpu_manager.go:414] "containerMap: removing container" podUID="465fd062-d546-453a-b343-95b93ae60c0d" containerName="revoke-keys" Nov 02 03:42:27.852940 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.852940 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465fd062-d546-453a-b343-95b93ae60c0d" containerName="revoke-keys" Nov 02 03:42:27.853143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.853130 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="465fd062-d546-453a-b343-95b93ae60c0d" containerName="revoke-keys" Nov 02 03:42:27.856644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.856624 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:27.917864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.917832 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q"] Nov 02 03:42:27.930761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.930733 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28w2\" (UniqueName: \"kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:27.930905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.930803 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:27.930905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.930867 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:27.931004 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.930985 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7wz\" (UniqueName: \"kubernetes.io/projected/465fd062-d546-453a-b343-95b93ae60c0d-kube-api-access-jw7wz\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:27.931038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.931009 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:27.931038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:27.931023 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465fd062-d546-453a-b343-95b93ae60c0d-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:28.031811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.031783 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.031811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.031814 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.032036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.031925 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28w2\" (UniqueName: \"kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.032509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.032485 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.034320 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.034300 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.087581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.087530 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28w2\" (UniqueName: \"kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2\") pod \"maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.173499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.173473 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:28.545203 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.545023 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q"] Nov 02 03:42:28.551498 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:42:28.551466 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1df167a_a952_41eb_aea0_c3916b6d18ee.slice/crio-3bfb4e0a8d054e055f08b87e84c6b29ee0330c6d193a6b5b7ee448f1ec985106 WatchSource:0}: Error finding container 3bfb4e0a8d054e055f08b87e84c6b29ee0330c6d193a6b5b7ee448f1ec985106: Status 404 returned error can't find the container with id 3bfb4e0a8d054e055f08b87e84c6b29ee0330c6d193a6b5b7ee448f1ec985106 Nov 02 03:42:28.555407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.555384 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" Nov 02 03:42:28.555541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.555383 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg" event={"ID":"465fd062-d546-453a-b343-95b93ae60c0d","Type":"ContainerDied","Data":"56bf43c0e44c0e81034cd0bc01ea8023a85bd78c23732efa7f47876231c1dda7"} Nov 02 03:42:28.555541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:28.555504 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56bf43c0e44c0e81034cd0bc01ea8023a85bd78c23732efa7f47876231c1dda7" Nov 02 03:42:29.563071 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:29.563034 4868 generic.go:356] "Generic (PLEG): container finished" podID="e1df167a-a952-41eb-aea0-c3916b6d18ee" containerID="b6a9150773faef9845bb782a2502aa11aa12b728ae8be5e842a758b4e084c0fd" exitCode=0 Nov 02 03:42:29.563474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:29.563081 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" event={"ID":"e1df167a-a952-41eb-aea0-c3916b6d18ee","Type":"ContainerDied","Data":"b6a9150773faef9845bb782a2502aa11aa12b728ae8be5e842a758b4e084c0fd"} Nov 02 03:42:29.563474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:29.563135 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" event={"ID":"e1df167a-a952-41eb-aea0-c3916b6d18ee","Type":"ContainerStarted","Data":"3bfb4e0a8d054e055f08b87e84c6b29ee0330c6d193a6b5b7ee448f1ec985106"} Nov 02 03:42:30.713382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.713347 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:30.866875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.866847 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2\" (UniqueName: \"kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2\") pod \"e1df167a-a952-41eb-aea0-c3916b6d18ee\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " Nov 02 03:42:30.867018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.866938 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca\") pod \"e1df167a-a952-41eb-aea0-c3916b6d18ee\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " Nov 02 03:42:30.867018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.866996 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca\") pod \"e1df167a-a952-41eb-aea0-c3916b6d18ee\" (UID: \"e1df167a-a952-41eb-aea0-c3916b6d18ee\") " Nov 02 03:42:30.867364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.867331 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca" pod "e1df167a-a952-41eb-aea0-c3916b6d18ee" (UID: "e1df167a-a952-41eb-aea0-c3916b6d18ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:42:30.868977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.868949 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca" pod "e1df167a-a952-41eb-aea0-c3916b6d18ee" (UID: "e1df167a-a952-41eb-aea0-c3916b6d18ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:42:30.869084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.869014 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2" pod "e1df167a-a952-41eb-aea0-c3916b6d18ee" (UID: "e1df167a-a952-41eb-aea0-c3916b6d18ee"). InnerVolumeSpecName "kube-api-access-s28w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:42:30.968115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.968088 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:30.968115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.968114 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1df167a-a952-41eb-aea0-c3916b6d18ee-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:30.968277 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:30.968126 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28w2\" (UniqueName: \"kubernetes.io/projected/e1df167a-a952-41eb-aea0-c3916b6d18ee-kube-api-access-s28w2\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:31.576175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:31.576147 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" Nov 02 03:42:31.576364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:31.576148 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q" event={"ID":"e1df167a-a952-41eb-aea0-c3916b6d18ee","Type":"ContainerDied","Data":"3bfb4e0a8d054e055f08b87e84c6b29ee0330c6d193a6b5b7ee448f1ec985106"} Nov 02 03:42:31.576364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:31.576264 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bfb4e0a8d054e055f08b87e84c6b29ee0330c6d193a6b5b7ee448f1ec985106" Nov 02 03:42:33.446052 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:42:33.446015 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:42:38.513483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:38.513452 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:42:38.520125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:38.520091 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:42:40.751222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.751192 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz"] Nov 02 03:42:40.751701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.751686 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e1df167a-a952-41eb-aea0-c3916b6d18ee" containerName="revoke-keys" Nov 02 03:42:40.751756 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.751702 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1df167a-a952-41eb-aea0-c3916b6d18ee" containerName="revoke-keys" Nov 02 03:42:40.751806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.751796 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1df167a-a952-41eb-aea0-c3916b6d18ee" containerName="revoke-keys" Nov 02 03:42:40.755045 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.755024 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.809324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.809296 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:42:40.859921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.859889 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:42:40.864101 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.863262 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz"] Nov 02 03:42:40.869298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.869270 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.869473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.869338 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.869540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.869473 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c78w\" (UniqueName: \"kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.970246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.970216 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.970472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.970374 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c78w\" (UniqueName: \"kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.970543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.970491 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.971147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.971122 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:40.973076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:40.973049 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:41.060829 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:41.060740 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c78w\" (UniqueName: \"kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w\") pod \"maas-api-revoke-sub-b1b1b27a-8dclz\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:41.071488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:41.071457 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:41.444107 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:41.444086 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz"] Nov 02 03:42:41.445301 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:42:41.445275 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef561126_48d2_465b_b1b6_3e1a2ec0dc4d.slice/crio-bcfef874ecff7bf448caa644408b0ecdcd8102bf2f0315782b95efc1a5a6e541 WatchSource:0}: Error finding container bcfef874ecff7bf448caa644408b0ecdcd8102bf2f0315782b95efc1a5a6e541: Status 404 returned error can't find the container with id bcfef874ecff7bf448caa644408b0ecdcd8102bf2f0315782b95efc1a5a6e541 Nov 02 03:42:41.655416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:41.653559 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" event={"ID":"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d","Type":"ContainerStarted","Data":"bcfef874ecff7bf448caa644408b0ecdcd8102bf2f0315782b95efc1a5a6e541"} Nov 02 03:42:42.680505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:42.680466 4868 generic.go:356] "Generic (PLEG): container finished" podID="ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" containerID="ff0ea610d3aa52a614163194e89111dd1f5b00f28b7eb95d52faa0ca804d9dd5" exitCode=0 Nov 02 03:42:42.680929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:42.680559 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" event={"ID":"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d","Type":"ContainerDied","Data":"ff0ea610d3aa52a614163194e89111dd1f5b00f28b7eb95d52faa0ca804d9dd5"} Nov 02 03:42:42.844639 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:42.844591 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl"] Nov 02 03:42:42.846776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:42.846753 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d563c-tnpjl"] Nov 02 03:42:43.835967 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.835939 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:43.903333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.903305 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca\") pod \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " Nov 02 03:42:43.903534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.903518 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w\" (UniqueName: \"kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w\") pod \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " Nov 02 03:42:43.903606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.903557 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca\") pod \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\" (UID: \"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d\") " Nov 02 03:42:43.903946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.903893 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca" pod "ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" (UID: "ef561126-48d2-465b-b1b6-3e1a2ec0dc4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:42:43.905509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.905485 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca" pod "ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" (UID: "ef561126-48d2-465b-b1b6-3e1a2ec0dc4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:42:43.905612 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:43.905583 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w" pod "ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" (UID: "ef561126-48d2-465b-b1b6-3e1a2ec0dc4d"). InnerVolumeSpecName "kube-api-access-8c78w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:42:44.005124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.005069 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c78w\" (UniqueName: \"kubernetes.io/projected/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-kube-api-access-8c78w\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:44.005124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.005093 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:44.005124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.005104 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:44.114338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.114314 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01c6772-1a9d-4af3-9fb8-6f607c86a94d" path="/var/lib/kubelet/pods/a01c6772-1a9d-4af3-9fb8-6f607c86a94d/volumes" Nov 02 03:42:44.694648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.694620 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" Nov 02 03:42:44.694819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.694653 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz" event={"ID":"ef561126-48d2-465b-b1b6-3e1a2ec0dc4d","Type":"ContainerDied","Data":"bcfef874ecff7bf448caa644408b0ecdcd8102bf2f0315782b95efc1a5a6e541"} Nov 02 03:42:44.694819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:44.694700 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcfef874ecff7bf448caa644408b0ecdcd8102bf2f0315782b95efc1a5a6e541" Nov 02 03:42:54.527464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.527429 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4"] Nov 02 03:42:54.527869 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.527856 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" containerName="revoke-keys" Nov 02 03:42:54.527915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.527870 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" containerName="revoke-keys" Nov 02 03:42:54.527992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.527983 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" containerName="revoke-keys" Nov 02 03:42:54.532452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.532430 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.586576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.586552 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4"] Nov 02 03:42:54.617692 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.617670 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.617819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.617705 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.617863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.617822 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8rb\" (UniqueName: \"kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.633399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.633373 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:42:54.633399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.633392 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:42:54.719043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.719015 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8rb\" (UniqueName: \"kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.719184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.719105 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.719184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.719133 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.719856 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.719834 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.721677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.721652 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.777944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.777887 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8rb\" (UniqueName: \"kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb\") pod \"maas-api-revoke-sub-a68e352d-zlwn4\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:54.845532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:54.845509 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:55.215873 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:55.215841 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4"] Nov 02 03:42:55.217636 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:42:55.217609 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f91ed18_412b_4c71_b694_ae9e0671ca2e.slice/crio-f9868d246c6bfb4100ba0e83a03e53d07605bd435e9c352f653279a0dffe8be1 WatchSource:0}: Error finding container f9868d246c6bfb4100ba0e83a03e53d07605bd435e9c352f653279a0dffe8be1: Status 404 returned error can't find the container with id f9868d246c6bfb4100ba0e83a03e53d07605bd435e9c352f653279a0dffe8be1 Nov 02 03:42:55.764387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:55.764329 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" event={"ID":"9f91ed18-412b-4c71-b694-ae9e0671ca2e","Type":"ContainerStarted","Data":"083520517c2501c6431ccb1d7be30802565d59ab5ca02bab58ecb2f7e8dee919"} Nov 02 03:42:55.764817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:55.764401 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" event={"ID":"9f91ed18-412b-4c71-b694-ae9e0671ca2e","Type":"ContainerStarted","Data":"f9868d246c6bfb4100ba0e83a03e53d07605bd435e9c352f653279a0dffe8be1"} Nov 02 03:42:55.879658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:55.879615 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" podStartSLOduration=1.879601762 podStartE2EDuration="1.879601762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:42:55.876494514 +0000 UTC m=+3317.872195469" watchObservedRunningTime="2026-11-02 03:42:55.879601762 +0000 UTC m=+3317.875302704" Nov 02 03:42:56.771744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:56.771705 4868 generic.go:356] "Generic (PLEG): container finished" podID="9f91ed18-412b-4c71-b694-ae9e0671ca2e" containerID="083520517c2501c6431ccb1d7be30802565d59ab5ca02bab58ecb2f7e8dee919" exitCode=0 Nov 02 03:42:56.772129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:56.771762 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" event={"ID":"9f91ed18-412b-4c71-b694-ae9e0671ca2e","Type":"ContainerDied","Data":"083520517c2501c6431ccb1d7be30802565d59ab5ca02bab58ecb2f7e8dee919"} Nov 02 03:42:57.511758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.511725 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v"] Nov 02 03:42:57.515103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.515085 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.572181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.572152 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v"] Nov 02 03:42:57.652278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.652254 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.652455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.652310 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.652455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.652423 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjw7r\" (UniqueName: \"kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.753765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.753732 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjw7r\" (UniqueName: \"kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.753872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.753825 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.753872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.753870 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.754467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.754445 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.756131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.756108 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.809782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.809753 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjw7r\" (UniqueName: \"kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r\") pod \"maas-api-revoke-sub-de7748e3-pmt8v\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.828571 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.828545 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:42:57.927392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:57.927372 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:58.058143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.058066 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca\") pod \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " Nov 02 03:42:58.058143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.058103 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb\" (UniqueName: \"kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb\") pod \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " Nov 02 03:42:58.058143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.058126 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca\") pod \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\" (UID: \"9f91ed18-412b-4c71-b694-ae9e0671ca2e\") " Nov 02 03:42:58.058561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.058526 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca" pod "9f91ed18-412b-4c71-b694-ae9e0671ca2e" (UID: "9f91ed18-412b-4c71-b694-ae9e0671ca2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:42:58.060191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.060168 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb" pod "9f91ed18-412b-4c71-b694-ae9e0671ca2e" (UID: "9f91ed18-412b-4c71-b694-ae9e0671ca2e"). InnerVolumeSpecName "kube-api-access-2f8rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:42:58.060688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.060669 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca" pod "9f91ed18-412b-4c71-b694-ae9e0671ca2e" (UID: "9f91ed18-412b-4c71-b694-ae9e0671ca2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:42:58.159741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.159715 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:58.159862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.159744 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f8rb\" (UniqueName: \"kubernetes.io/projected/9f91ed18-412b-4c71-b694-ae9e0671ca2e-kube-api-access-2f8rb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:58.159862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.159757 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f91ed18-412b-4c71-b694-ae9e0671ca2e-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:42:58.212891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.212868 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v"] Nov 02 03:42:58.214900 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:42:58.214867 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafa5e25_bc39_4f3b_ac32_9c329e2f98a6.slice/crio-bc81800d563d31c298fce88dad323bfe2ae734058019902a0cc6905d2dd1a154 WatchSource:0}: Error finding container bc81800d563d31c298fce88dad323bfe2ae734058019902a0cc6905d2dd1a154: Status 404 returned error can't find the container with id bc81800d563d31c298fce88dad323bfe2ae734058019902a0cc6905d2dd1a154 Nov 02 03:42:58.790876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.790845 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" event={"ID":"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6","Type":"ContainerStarted","Data":"c8489261cb0d8075de53092f5800411098459d68b40bed024b24174cb9be0381"} Nov 02 03:42:58.791024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.790905 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" event={"ID":"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6","Type":"ContainerStarted","Data":"bc81800d563d31c298fce88dad323bfe2ae734058019902a0cc6905d2dd1a154"} Nov 02 03:42:58.792213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.792194 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" Nov 02 03:42:58.792313 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.792218 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4" event={"ID":"9f91ed18-412b-4c71-b694-ae9e0671ca2e","Type":"ContainerDied","Data":"f9868d246c6bfb4100ba0e83a03e53d07605bd435e9c352f653279a0dffe8be1"} Nov 02 03:42:58.792313 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.792251 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9868d246c6bfb4100ba0e83a03e53d07605bd435e9c352f653279a0dffe8be1" Nov 02 03:42:58.904675 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:58.904620 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" podStartSLOduration=1.904602062 podStartE2EDuration="1.904602062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:42:58.901714707 +0000 UTC m=+3320.897415665" watchObservedRunningTime="2026-11-02 03:42:58.904602062 +0000 UTC m=+3320.900303005" Nov 02 03:42:59.799431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:59.799400 4868 generic.go:356] "Generic (PLEG): container finished" podID="cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" containerID="c8489261cb0d8075de53092f5800411098459d68b40bed024b24174cb9be0381" exitCode=0 Nov 02 03:42:59.799602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:42:59.799448 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" event={"ID":"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6","Type":"ContainerDied","Data":"c8489261cb0d8075de53092f5800411098459d68b40bed024b24174cb9be0381"} Nov 02 03:43:00.975041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:00.975016 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:43:01.104200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.104171 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca\") pod \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " Nov 02 03:43:01.104343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.104220 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r\" (UniqueName: \"kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r\") pod \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " Nov 02 03:43:01.104343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.104271 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca\") pod \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\" (UID: \"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6\") " Nov 02 03:43:01.104627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.104600 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca" pod "cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" (UID: "cafa5e25-bc39-4f3b-ac32-9c329e2f98a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:01.104811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.104792 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:01.106371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.106324 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r" pod "cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" (UID: "cafa5e25-bc39-4f3b-ac32-9c329e2f98a6"). InnerVolumeSpecName "kube-api-access-bjw7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:01.106721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.106690 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca" pod "cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" (UID: "cafa5e25-bc39-4f3b-ac32-9c329e2f98a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:01.206305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.206258 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjw7r\" (UniqueName: \"kubernetes.io/projected/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-kube-api-access-bjw7r\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:01.206305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.206297 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:01.829491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.829457 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" event={"ID":"cafa5e25-bc39-4f3b-ac32-9c329e2f98a6","Type":"ContainerDied","Data":"bc81800d563d31c298fce88dad323bfe2ae734058019902a0cc6905d2dd1a154"} Nov 02 03:43:01.829491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.829481 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v" Nov 02 03:43:01.829719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:01.829507 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc81800d563d31c298fce88dad323bfe2ae734058019902a0cc6905d2dd1a154" Nov 02 03:43:03.475435 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:43:03.475391 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:43:05.381798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.381761 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27"] Nov 02 03:43:05.382221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.382208 4868 cpu_manager.go:414] "containerMap: removing container" podUID="cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" containerName="revoke-keys" Nov 02 03:43:05.382272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.382222 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" containerName="revoke-keys" Nov 02 03:43:05.382272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.382256 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9f91ed18-412b-4c71-b694-ae9e0671ca2e" containerName="revoke-keys" Nov 02 03:43:05.382272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.382261 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f91ed18-412b-4c71-b694-ae9e0671ca2e" containerName="revoke-keys" Nov 02 03:43:05.382402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.382362 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f91ed18-412b-4c71-b694-ae9e0671ca2e" containerName="revoke-keys" Nov 02 03:43:05.382402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.382371 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" containerName="revoke-keys" Nov 02 03:43:05.385880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.385863 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.459818 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.458111 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27"] Nov 02 03:43:05.474822 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.474791 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.474965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.474838 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsbl\" (UniqueName: \"kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.474965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.474953 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.493634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.493609 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:43:05.493740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.493653 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:43:05.576018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.575986 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsbl\" (UniqueName: \"kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.576182 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.576111 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.576245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.576221 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.576863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.576843 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.578467 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.578450 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.654417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.647985 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsbl\" (UniqueName: \"kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl\") pod \"maas-api-revoke-sub-ef2e5a37-hdh27\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:05.698533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:05.698504 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:06.091721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.091690 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27"] Nov 02 03:43:06.093333 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:06.093302 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d1404a_7697_4309_ae4e_edddf5d602a1.slice/crio-72864ea68ec57c9b6cc53ba700883de41c6d1d2bae880d89b50a6c53e26a61d1 WatchSource:0}: Error finding container 72864ea68ec57c9b6cc53ba700883de41c6d1d2bae880d89b50a6c53e26a61d1: Status 404 returned error can't find the container with id 72864ea68ec57c9b6cc53ba700883de41c6d1d2bae880d89b50a6c53e26a61d1 Nov 02 03:43:06.325241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.325209 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th"] Nov 02 03:43:06.362925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.362893 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.385293 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.385262 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th"] Nov 02 03:43:06.385747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.385717 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.385868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.385849 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ckf\" (UniqueName: \"kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.385941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.385922 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.486712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.486685 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.486867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.486819 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ckf\" (UniqueName: \"kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.486867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.486855 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.487484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.487465 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.488895 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.488874 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.557465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.557439 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ckf\" (UniqueName: \"kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf\") pod \"maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.676239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.676162 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:06.885217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.885177 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" event={"ID":"67d1404a-7697-4309-ae4e-edddf5d602a1","Type":"ContainerStarted","Data":"16595fe2438b5e31d0eb0748ee09ca2ca29c3d7c3ba2636b6cf07024e9c449ff"} Nov 02 03:43:06.885399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:06.885241 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" event={"ID":"67d1404a-7697-4309-ae4e-edddf5d602a1","Type":"ContainerStarted","Data":"72864ea68ec57c9b6cc53ba700883de41c6d1d2bae880d89b50a6c53e26a61d1"} Nov 02 03:43:07.000868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:07.000740 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" podStartSLOduration=2.000721174 podStartE2EDuration="2.000721174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:06.995975451 +0000 UTC m=+3328.991676395" watchObservedRunningTime="2026-11-02 03:43:07.000721174 +0000 UTC m=+3328.996422119" Nov 02 03:43:07.050189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:07.050161 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th"] Nov 02 03:43:07.099568 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:07.099520 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fb0115_6e81_4075_ac5a_91db7c60a68c.slice/crio-7931a577f8def76586da7ba06613b16179bfafb8bfdfe006c361d57f330fc8b1 WatchSource:0}: Error finding container 7931a577f8def76586da7ba06613b16179bfafb8bfdfe006c361d57f330fc8b1: Status 404 returned error can't find the container with id 7931a577f8def76586da7ba06613b16179bfafb8bfdfe006c361d57f330fc8b1 Nov 02 03:43:07.894007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:07.893966 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" event={"ID":"b2fb0115-6e81-4075-ac5a-91db7c60a68c","Type":"ContainerStarted","Data":"d4757c40788ea9e9616a93c9270d825b09dc15de68623cc6a86646a9043ab5ed"} Nov 02 03:43:07.894565 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:07.894036 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" event={"ID":"b2fb0115-6e81-4075-ac5a-91db7c60a68c","Type":"ContainerStarted","Data":"7931a577f8def76586da7ba06613b16179bfafb8bfdfe006c361d57f330fc8b1"} Nov 02 03:43:07.895898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:07.895864 4868 generic.go:356] "Generic (PLEG): container finished" podID="67d1404a-7697-4309-ae4e-edddf5d602a1" containerID="16595fe2438b5e31d0eb0748ee09ca2ca29c3d7c3ba2636b6cf07024e9c449ff" exitCode=0 Nov 02 03:43:07.896050 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:07.895952 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" event={"ID":"67d1404a-7697-4309-ae4e-edddf5d602a1","Type":"ContainerDied","Data":"16595fe2438b5e31d0eb0748ee09ca2ca29c3d7c3ba2636b6cf07024e9c449ff"} Nov 02 03:43:08.009809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.009758 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" podStartSLOduration=2.009744122 podStartE2EDuration="2.009744122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:08.005960298 +0000 UTC m=+3330.001661241" watchObservedRunningTime="2026-11-02 03:43:08.009744122 +0000 UTC m=+3330.005445057" Nov 02 03:43:08.335826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.335750 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl"] Nov 02 03:43:08.339631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.339615 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.394809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.394783 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl"] Nov 02 03:43:08.406771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.406747 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t924g\" (UniqueName: \"kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.406879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.406825 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.406933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.406890 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.508163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.508125 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.508346 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.508188 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t924g\" (UniqueName: \"kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.508346 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.508295 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.509069 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.509040 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.510712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.510685 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.564088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.564058 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t924g\" (UniqueName: \"kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g\") pod \"maas-api-revoke-sub-daae74a4-kmgtl\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.653112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.653089 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:08.903581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.903486 4868 generic.go:356] "Generic (PLEG): container finished" podID="b2fb0115-6e81-4075-ac5a-91db7c60a68c" containerID="d4757c40788ea9e9616a93c9270d825b09dc15de68623cc6a86646a9043ab5ed" exitCode=0 Nov 02 03:43:08.903581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:08.903548 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" event={"ID":"b2fb0115-6e81-4075-ac5a-91db7c60a68c","Type":"ContainerDied","Data":"d4757c40788ea9e9616a93c9270d825b09dc15de68623cc6a86646a9043ab5ed"} Nov 02 03:43:09.028209 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:09.028172 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec43308_3716_4db1_a8ac_5b3b30f041bd.slice/crio-6747e773be280207f1c3dadbb506222b36bbfba0033c0973f5c6d656d15dcfbd WatchSource:0}: Error finding container 6747e773be280207f1c3dadbb506222b36bbfba0033c0973f5c6d656d15dcfbd: Status 404 returned error can't find the container with id 6747e773be280207f1c3dadbb506222b36bbfba0033c0973f5c6d656d15dcfbd Nov 02 03:43:09.034911 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.034884 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl"] Nov 02 03:43:09.157084 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.157031 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:09.216445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.216410 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca\") pod \"67d1404a-7697-4309-ae4e-edddf5d602a1\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " Nov 02 03:43:09.216593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.216539 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca\") pod \"67d1404a-7697-4309-ae4e-edddf5d602a1\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " Nov 02 03:43:09.216658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.216641 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl\" (UniqueName: \"kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl\") pod \"67d1404a-7697-4309-ae4e-edddf5d602a1\" (UID: \"67d1404a-7697-4309-ae4e-edddf5d602a1\") " Nov 02 03:43:09.216951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.216922 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca" pod "67d1404a-7697-4309-ae4e-edddf5d602a1" (UID: "67d1404a-7697-4309-ae4e-edddf5d602a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:09.218651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.218627 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca" pod "67d1404a-7697-4309-ae4e-edddf5d602a1" (UID: "67d1404a-7697-4309-ae4e-edddf5d602a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:09.218828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.218806 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl" pod "67d1404a-7697-4309-ae4e-edddf5d602a1" (UID: "67d1404a-7697-4309-ae4e-edddf5d602a1"). InnerVolumeSpecName "kube-api-access-crsbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:09.318086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.318043 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:09.318086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.318089 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67d1404a-7697-4309-ae4e-edddf5d602a1-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:09.318575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.318109 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crsbl\" (UniqueName: \"kubernetes.io/projected/67d1404a-7697-4309-ae4e-edddf5d602a1-kube-api-access-crsbl\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:09.911117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.911077 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" event={"ID":"fec43308-3716-4db1-a8ac-5b3b30f041bd","Type":"ContainerStarted","Data":"8d0fd5cd42f4913b6a09c16a822b1744a0d1c22168a72e78d563670da55bf635"} Nov 02 03:43:09.913653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.911142 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" event={"ID":"fec43308-3716-4db1-a8ac-5b3b30f041bd","Type":"ContainerStarted","Data":"6747e773be280207f1c3dadbb506222b36bbfba0033c0973f5c6d656d15dcfbd"} Nov 02 03:43:09.913653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.912718 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" Nov 02 03:43:09.913653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.912755 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27" event={"ID":"67d1404a-7697-4309-ae4e-edddf5d602a1","Type":"ContainerDied","Data":"72864ea68ec57c9b6cc53ba700883de41c6d1d2bae880d89b50a6c53e26a61d1"} Nov 02 03:43:09.913653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.912803 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72864ea68ec57c9b6cc53ba700883de41c6d1d2bae880d89b50a6c53e26a61d1" Nov 02 03:43:09.985453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.981855 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 03:43:09.985453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:09.981908 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 03:43:10.026745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.026698 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" podStartSLOduration=2.026684423 podStartE2EDuration="2.026684423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:10.024478062 +0000 UTC m=+3332.020179006" watchObservedRunningTime="2026-11-02 03:43:10.026684423 +0000 UTC m=+3332.022385398" Nov 02 03:43:10.102504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.102469 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:10.230875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.230792 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca\") pod \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " Nov 02 03:43:10.231040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.230912 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca\") pod \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " Nov 02 03:43:10.231040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.230965 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf\" (UniqueName: \"kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf\") pod \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\" (UID: \"b2fb0115-6e81-4075-ac5a-91db7c60a68c\") " Nov 02 03:43:10.231206 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.231176 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca" pod "b2fb0115-6e81-4075-ac5a-91db7c60a68c" (UID: "b2fb0115-6e81-4075-ac5a-91db7c60a68c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:10.231841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.231591 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:10.233711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.233670 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf" pod "b2fb0115-6e81-4075-ac5a-91db7c60a68c" (UID: "b2fb0115-6e81-4075-ac5a-91db7c60a68c"). InnerVolumeSpecName "kube-api-access-29ckf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:10.233858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.233826 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca" pod "b2fb0115-6e81-4075-ac5a-91db7c60a68c" (UID: "b2fb0115-6e81-4075-ac5a-91db7c60a68c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:10.332386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.332344 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2fb0115-6e81-4075-ac5a-91db7c60a68c-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:10.332386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.332385 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ckf\" (UniqueName: \"kubernetes.io/projected/b2fb0115-6e81-4075-ac5a-91db7c60a68c-kube-api-access-29ckf\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:10.920106 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.920066 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" Nov 02 03:43:10.920106 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.920090 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th" event={"ID":"b2fb0115-6e81-4075-ac5a-91db7c60a68c","Type":"ContainerDied","Data":"7931a577f8def76586da7ba06613b16179bfafb8bfdfe006c361d57f330fc8b1"} Nov 02 03:43:10.920632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.920144 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7931a577f8def76586da7ba06613b16179bfafb8bfdfe006c361d57f330fc8b1" Nov 02 03:43:10.921905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.921882 4868 generic.go:356] "Generic (PLEG): container finished" podID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerID="730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae" exitCode=0 Nov 02 03:43:10.922013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.921959 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" event={"ID":"23b7191e-00d3-4cae-8883-2ca020f5467b","Type":"ContainerDied","Data":"730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae"} Nov 02 03:43:10.922403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.922379 4868 scope.go:122] "RemoveContainer" containerID="730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae" Nov 02 03:43:10.923887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.923866 4868 generic.go:356] "Generic (PLEG): container finished" podID="fec43308-3716-4db1-a8ac-5b3b30f041bd" containerID="8d0fd5cd42f4913b6a09c16a822b1744a0d1c22168a72e78d563670da55bf635" exitCode=0 Nov 02 03:43:10.923999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.923930 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" event={"ID":"fec43308-3716-4db1-a8ac-5b3b30f041bd","Type":"ContainerDied","Data":"8d0fd5cd42f4913b6a09c16a822b1744a0d1c22168a72e78d563670da55bf635"} Nov 02 03:43:10.924082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.924018 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:43:10.924178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.924161 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:43:10.924239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:10.924220 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:43:11.935386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:11.932226 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" event={"ID":"23b7191e-00d3-4cae-8883-2ca020f5467b","Type":"ContainerStarted","Data":"56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2"} Nov 02 03:43:12.156783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.156748 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:12.160188 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.160158 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca\") pod \"fec43308-3716-4db1-a8ac-5b3b30f041bd\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " Nov 02 03:43:12.160381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.160255 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g\" (UniqueName: \"kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g\") pod \"fec43308-3716-4db1-a8ac-5b3b30f041bd\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " Nov 02 03:43:12.160381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.160289 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca\") pod \"fec43308-3716-4db1-a8ac-5b3b30f041bd\" (UID: \"fec43308-3716-4db1-a8ac-5b3b30f041bd\") " Nov 02 03:43:12.160605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.160576 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca" pod "fec43308-3716-4db1-a8ac-5b3b30f041bd" (UID: "fec43308-3716-4db1-a8ac-5b3b30f041bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:12.160789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.160773 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:12.163666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.163643 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca" pod "fec43308-3716-4db1-a8ac-5b3b30f041bd" (UID: "fec43308-3716-4db1-a8ac-5b3b30f041bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:12.165789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.165763 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g" pod "fec43308-3716-4db1-a8ac-5b3b30f041bd" (UID: "fec43308-3716-4db1-a8ac-5b3b30f041bd"). InnerVolumeSpecName "kube-api-access-t924g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:12.262430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.262285 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t924g\" (UniqueName: \"kubernetes.io/projected/fec43308-3716-4db1-a8ac-5b3b30f041bd-kube-api-access-t924g\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:12.262430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.262327 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec43308-3716-4db1-a8ac-5b3b30f041bd-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:12.669373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.668947 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:43:12.669373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.669160 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 03:43:12.669373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.669235 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 03:43:12.961607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.961528 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" Nov 02 03:43:12.961607 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.961565 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl" event={"ID":"fec43308-3716-4db1-a8ac-5b3b30f041bd","Type":"ContainerDied","Data":"6747e773be280207f1c3dadbb506222b36bbfba0033c0973f5c6d656d15dcfbd"} Nov 02 03:43:12.962029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:12.961611 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6747e773be280207f1c3dadbb506222b36bbfba0033c0973f5c6d656d15dcfbd" Nov 02 03:43:13.661360 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:13.661314 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 03:43:13.661617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:13.661411 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 03:43:14.451377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:14.450475 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q"] Nov 02 03:43:14.483681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:14.483648 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-167a01-64z9q"] Nov 02 03:43:14.661458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:14.661420 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 03:43:14.661640 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:14.661501 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 03:43:15.661874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:15.661831 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 03:43:15.662261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:15.661894 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 03:43:16.117437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:16.117398 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1df167a-a952-41eb-aea0-c3916b6d18ee" path="/var/lib/kubelet/pods/e1df167a-a952-41eb-aea0-c3916b6d18ee/volumes" Nov 02 03:43:16.661992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:16.661960 4868 patch_prober.go:29] interesting pod/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 02 03:43:16.662399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:16.662019 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 02 03:43:17.666121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:17.666095 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:43:17.666529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:17.666471 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:43:17.666979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:17.666962 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:43:21.009660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.009626 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk"] Nov 02 03:43:21.010179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.009873 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" containerID="cri-o://3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8" gracePeriod=30 Nov 02 03:43:21.552902 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.552867 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975"] Nov 02 03:43:21.553319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553306 4868 cpu_manager.go:414] "containerMap: removing container" podUID="fec43308-3716-4db1-a8ac-5b3b30f041bd" containerName="revoke-keys" Nov 02 03:43:21.553400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553320 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec43308-3716-4db1-a8ac-5b3b30f041bd" containerName="revoke-keys" Nov 02 03:43:21.553400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553331 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b2fb0115-6e81-4075-ac5a-91db7c60a68c" containerName="revoke-keys" Nov 02 03:43:21.553400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553337 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fb0115-6e81-4075-ac5a-91db7c60a68c" containerName="revoke-keys" Nov 02 03:43:21.553400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553383 4868 cpu_manager.go:414] "containerMap: removing container" podUID="67d1404a-7697-4309-ae4e-edddf5d602a1" containerName="revoke-keys" Nov 02 03:43:21.553400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553389 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d1404a-7697-4309-ae4e-edddf5d602a1" containerName="revoke-keys" Nov 02 03:43:21.553558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553481 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec43308-3716-4db1-a8ac-5b3b30f041bd" containerName="revoke-keys" Nov 02 03:43:21.553558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553490 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fb0115-6e81-4075-ac5a-91db7c60a68c" containerName="revoke-keys" Nov 02 03:43:21.553558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.553497 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d1404a-7697-4309-ae4e-edddf5d602a1" containerName="revoke-keys" Nov 02 03:43:21.556711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.556694 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.609233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.609182 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:43:21.609443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.609266 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:43:21.613504 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.613481 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975"] Nov 02 03:43:21.690124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.690091 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.690337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.690144 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.690337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.690168 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4mm\" (UniqueName: \"kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.746713 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.746673 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj"] Nov 02 03:43:21.751626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.751599 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.791738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.791703 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfv2d\" (UniqueName: \"kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.791941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.791780 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.791941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.791886 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.791941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.791923 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4mm\" (UniqueName: \"kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.792116 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.791983 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.792116 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.792037 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.792655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.792637 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.794141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.794121 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.803108 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.803046 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj"] Nov 02 03:43:21.848085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.848055 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4mm\" (UniqueName: \"kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm\") pod \"maas-api-revoke-sub-0b8eb801-86975\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.869440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.869413 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:21.893704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.893670 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.893836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.893756 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.893836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.893821 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfv2d\" (UniqueName: \"kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.894732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.894705 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.896365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.896329 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:21.963239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:21.963172 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfv2d\" (UniqueName: \"kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d\") pod \"maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:22.079651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:22.079568 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:22.258551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:22.258518 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975"] Nov 02 03:43:22.259702 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:22.259678 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106e6e43_0ec0_4353_9212_2596c81cefe3.slice/crio-744be8fc0ca2f570fbd346a9d87e2a85dd22c77d7711adf2034a2b3bbc95f260 WatchSource:0}: Error finding container 744be8fc0ca2f570fbd346a9d87e2a85dd22c77d7711adf2034a2b3bbc95f260: Status 404 returned error can't find the container with id 744be8fc0ca2f570fbd346a9d87e2a85dd22c77d7711adf2034a2b3bbc95f260 Nov 02 03:43:22.456432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:22.456409 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj"] Nov 02 03:43:22.457908 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:22.457873 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f12570_183f_4c03_9452_1de8eba4be7b.slice/crio-c64d7eeb81acad078003aee6509e95988ecbc057e64948028dc01fd57e08e3a4 WatchSource:0}: Error finding container c64d7eeb81acad078003aee6509e95988ecbc057e64948028dc01fd57e08e3a4: Status 404 returned error can't find the container with id c64d7eeb81acad078003aee6509e95988ecbc057e64948028dc01fd57e08e3a4 Nov 02 03:43:23.030113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:23.030014 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" event={"ID":"90f12570-183f-4c03-9452-1de8eba4be7b","Type":"ContainerStarted","Data":"3f8ad68350c70689bed2ed2b71cd1582e08b200b768dae3afa01d3734996d080"} Nov 02 03:43:23.030113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:23.030096 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" event={"ID":"90f12570-183f-4c03-9452-1de8eba4be7b","Type":"ContainerStarted","Data":"c64d7eeb81acad078003aee6509e95988ecbc057e64948028dc01fd57e08e3a4"} Nov 02 03:43:23.031666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:23.031638 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" event={"ID":"106e6e43-0ec0-4353-9212-2596c81cefe3","Type":"ContainerStarted","Data":"e00a9a2cfb2032848f55a94bea42ecd52cfbcf76c6bcc04197a17d6e785b1ad8"} Nov 02 03:43:23.031774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:23.031686 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" event={"ID":"106e6e43-0ec0-4353-9212-2596c81cefe3","Type":"ContainerStarted","Data":"744be8fc0ca2f570fbd346a9d87e2a85dd22c77d7711adf2034a2b3bbc95f260"} Nov 02 03:43:23.145125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:23.145078 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" podStartSLOduration=2.145065931 podStartE2EDuration="2.145065931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:23.141512132 +0000 UTC m=+3345.137213073" watchObservedRunningTime="2026-11-02 03:43:23.145065931 +0000 UTC m=+3345.140766889" Nov 02 03:43:23.255942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:23.255887 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" podStartSLOduration=2.255871989 podStartE2EDuration="2.255871989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:43:23.252772017 +0000 UTC m=+3345.248472959" watchObservedRunningTime="2026-11-02 03:43:23.255871989 +0000 UTC m=+3345.251572931" Nov 02 03:43:24.039682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.039648 4868 generic.go:356] "Generic (PLEG): container finished" podID="90f12570-183f-4c03-9452-1de8eba4be7b" containerID="3f8ad68350c70689bed2ed2b71cd1582e08b200b768dae3afa01d3734996d080" exitCode=0 Nov 02 03:43:24.039873 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.039732 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" event={"ID":"90f12570-183f-4c03-9452-1de8eba4be7b","Type":"ContainerDied","Data":"3f8ad68350c70689bed2ed2b71cd1582e08b200b768dae3afa01d3734996d080"} Nov 02 03:43:24.041376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.041341 4868 generic.go:356] "Generic (PLEG): container finished" podID="106e6e43-0ec0-4353-9212-2596c81cefe3" containerID="e00a9a2cfb2032848f55a94bea42ecd52cfbcf76c6bcc04197a17d6e785b1ad8" exitCode=0 Nov 02 03:43:24.041496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.041422 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" event={"ID":"106e6e43-0ec0-4353-9212-2596c81cefe3","Type":"ContainerDied","Data":"e00a9a2cfb2032848f55a94bea42ecd52cfbcf76c6bcc04197a17d6e785b1ad8"} Nov 02 03:43:24.514118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.514088 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l"] Nov 02 03:43:24.517867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.517846 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.573981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.573952 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l"] Nov 02 03:43:24.625846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.625814 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.625979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.625855 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.626025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.625986 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r82t\" (UniqueName: \"kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.726835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.726798 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.726976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.726842 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.726976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.726921 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r82t\" (UniqueName: \"kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.727490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.727467 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.729194 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.729171 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.782971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.782916 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r82t\" (UniqueName: \"kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t\") pod \"maas-api-revoke-sub-ec06e73e-phn7l\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:24.830964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:24.830938 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:25.191428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.191395 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:25.209716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.209693 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:25.223535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.223513 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l"] Nov 02 03:43:25.224794 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:25.224768 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda072928c_e8d8_425d_a3ad_9efe9db46e27.slice/crio-2c49fd0ba8efd36cc12475839062dd30acc402e1a1b364612bb344ea13153c43 WatchSource:0}: Error finding container 2c49fd0ba8efd36cc12475839062dd30acc402e1a1b364612bb344ea13153c43: Status 404 returned error can't find the container with id 2c49fd0ba8efd36cc12475839062dd30acc402e1a1b364612bb344ea13153c43 Nov 02 03:43:25.333866 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.333803 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca\") pod \"90f12570-183f-4c03-9452-1de8eba4be7b\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " Nov 02 03:43:25.334002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.333878 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm\" (UniqueName: \"kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm\") pod \"106e6e43-0ec0-4353-9212-2596c81cefe3\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " Nov 02 03:43:25.334002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.333989 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca\") pod \"106e6e43-0ec0-4353-9212-2596c81cefe3\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " Nov 02 03:43:25.334114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.334025 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca\") pod \"106e6e43-0ec0-4353-9212-2596c81cefe3\" (UID: \"106e6e43-0ec0-4353-9212-2596c81cefe3\") " Nov 02 03:43:25.334114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.334075 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d\" (UniqueName: \"kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d\") pod \"90f12570-183f-4c03-9452-1de8eba4be7b\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " Nov 02 03:43:25.334220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.334169 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca\") pod \"90f12570-183f-4c03-9452-1de8eba4be7b\" (UID: \"90f12570-183f-4c03-9452-1de8eba4be7b\") " Nov 02 03:43:25.334448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.334410 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca" pod "106e6e43-0ec0-4353-9212-2596c81cefe3" (UID: "106e6e43-0ec0-4353-9212-2596c81cefe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:25.334537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.334520 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:25.334820 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.334788 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca" pod "90f12570-183f-4c03-9452-1de8eba4be7b" (UID: "90f12570-183f-4c03-9452-1de8eba4be7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:25.336443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.336419 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d" pod "90f12570-183f-4c03-9452-1de8eba4be7b" (UID: "90f12570-183f-4c03-9452-1de8eba4be7b"). InnerVolumeSpecName "kube-api-access-pfv2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:25.336558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.336536 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm" pod "106e6e43-0ec0-4353-9212-2596c81cefe3" (UID: "106e6e43-0ec0-4353-9212-2596c81cefe3"). InnerVolumeSpecName "kube-api-access-nh4mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:25.336790 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.336774 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca" pod "90f12570-183f-4c03-9452-1de8eba4be7b" (UID: "90f12570-183f-4c03-9452-1de8eba4be7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:25.337409 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.337389 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca" pod "106e6e43-0ec0-4353-9212-2596c81cefe3" (UID: "106e6e43-0ec0-4353-9212-2596c81cefe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:25.435147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.435121 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:25.435147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.435145 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4mm\" (UniqueName: \"kubernetes.io/projected/106e6e43-0ec0-4353-9212-2596c81cefe3-kube-api-access-nh4mm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:25.435400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.435157 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106e6e43-0ec0-4353-9212-2596c81cefe3-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:25.435400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.435168 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfv2d\" (UniqueName: \"kubernetes.io/projected/90f12570-183f-4c03-9452-1de8eba4be7b-kube-api-access-pfv2d\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:25.435400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:25.435177 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f12570-183f-4c03-9452-1de8eba4be7b-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.054865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.054829 4868 generic.go:356] "Generic (PLEG): container finished" podID="a072928c-e8d8-425d-a3ad-9efe9db46e27" containerID="b031c21ae6953adc651b350e8cb1a9b674f1a9fe9ba925d88d835778e45d8405" exitCode=0 Nov 02 03:43:26.055238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.054910 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" event={"ID":"a072928c-e8d8-425d-a3ad-9efe9db46e27","Type":"ContainerDied","Data":"b031c21ae6953adc651b350e8cb1a9b674f1a9fe9ba925d88d835778e45d8405"} Nov 02 03:43:26.055238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.055011 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" event={"ID":"a072928c-e8d8-425d-a3ad-9efe9db46e27","Type":"ContainerStarted","Data":"2c49fd0ba8efd36cc12475839062dd30acc402e1a1b364612bb344ea13153c43"} Nov 02 03:43:26.056386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.056344 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" Nov 02 03:43:26.056386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.056372 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj" event={"ID":"90f12570-183f-4c03-9452-1de8eba4be7b","Type":"ContainerDied","Data":"c64d7eeb81acad078003aee6509e95988ecbc057e64948028dc01fd57e08e3a4"} Nov 02 03:43:26.056569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.056413 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64d7eeb81acad078003aee6509e95988ecbc057e64948028dc01fd57e08e3a4" Nov 02 03:43:26.057685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.057666 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" event={"ID":"106e6e43-0ec0-4353-9212-2596c81cefe3","Type":"ContainerDied","Data":"744be8fc0ca2f570fbd346a9d87e2a85dd22c77d7711adf2034a2b3bbc95f260"} Nov 02 03:43:26.057685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.057682 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975" Nov 02 03:43:26.057903 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.057698 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744be8fc0ca2f570fbd346a9d87e2a85dd22c77d7711adf2034a2b3bbc95f260" Nov 02 03:43:26.292199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.292177 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:43:26.344711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344682 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.344885 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344726 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.344885 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344764 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.344885 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344796 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.344885 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344839 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.345098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344925 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.345098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.344983 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:43:26.345098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345048 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.345254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345123 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.345254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345156 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy\") pod \"28f3df67-086d-46a5-85cf-fbc65b88b874\" (UID: \"28f3df67-086d-46a5-85cf-fbc65b88b874\") " Nov 02 03:43:26.345254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345119 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:43:26.345254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345213 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:26.345254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345219 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:43:26.345552 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345374 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:43:26.345611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345594 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.345670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345613 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.345670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345629 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.345670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345644 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.345670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.345657 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f3df67-086d-46a5-85cf-fbc65b88b874-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.347165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.347141 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "kube-api-access-brfnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:26.347165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.347159 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:26.347304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.347169 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:43:26.347478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.347459 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy" pod "28f3df67-086d-46a5-85cf-fbc65b88b874" (UID: "28f3df67-086d-46a5-85cf-fbc65b88b874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:43:26.446701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.446666 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.446701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.446694 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.446701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.446704 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfnv\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-kube-api-access-brfnv\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:26.446966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:26.446735 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f3df67-086d-46a5-85cf-fbc65b88b874-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:27.066185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.066108 4868 generic.go:356] "Generic (PLEG): container finished" podID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerID="3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8" exitCode=0 Nov 02 03:43:27.066185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.066156 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" Nov 02 03:43:27.066694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.066193 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" event={"ID":"28f3df67-086d-46a5-85cf-fbc65b88b874","Type":"ContainerDied","Data":"3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8"} Nov 02 03:43:27.066694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.066244 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk" event={"ID":"28f3df67-086d-46a5-85cf-fbc65b88b874","Type":"ContainerDied","Data":"dbe5c1ccab65d64b404938ae7298ffecbde500f10624411cca22de3b5114af6a"} Nov 02 03:43:27.066694 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.066267 4868 scope.go:122] "RemoveContainer" containerID="3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8" Nov 02 03:43:27.076515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.076495 4868 scope.go:122] "RemoveContainer" containerID="224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e" Nov 02 03:43:27.086256 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.086240 4868 scope.go:122] "RemoveContainer" containerID="3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8" Nov 02 03:43:27.086560 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:43:27.086537 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8\": container with ID starting with 3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8 not found: ID does not exist" containerID="3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8" Nov 02 03:43:27.086655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.086567 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8"} err="failed to get container status \"3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8\": rpc error: code = NotFound desc = could not find container \"3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8\": container with ID starting with 3e5caf447e997f812c187b59c9e4e15eb6e87f90164de0f38f6dfcd6e3fe96b8 not found: ID does not exist" Nov 02 03:43:27.086655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.086584 4868 scope.go:122] "RemoveContainer" containerID="224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e" Nov 02 03:43:27.086830 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:43:27.086815 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e\": container with ID starting with 224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e not found: ID does not exist" containerID="224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e" Nov 02 03:43:27.086876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.086835 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e"} err="failed to get container status \"224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e\": rpc error: code = NotFound desc = could not find container \"224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e\": container with ID starting with 224c9139db413a2a5607ff8e4dde43a4d2b5fe71a46208aea41160c51bcdad7e not found: ID does not exist" Nov 02 03:43:27.200277 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.200251 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:27.230932 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.230902 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk"] Nov 02 03:43:27.254426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.254399 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca\") pod \"a072928c-e8d8-425d-a3ad-9efe9db46e27\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " Nov 02 03:43:27.254535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.254448 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t\" (UniqueName: \"kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t\") pod \"a072928c-e8d8-425d-a3ad-9efe9db46e27\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " Nov 02 03:43:27.254598 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.254578 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca\") pod \"a072928c-e8d8-425d-a3ad-9efe9db46e27\" (UID: \"a072928c-e8d8-425d-a3ad-9efe9db46e27\") " Nov 02 03:43:27.254716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.254688 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca" pod "a072928c-e8d8-425d-a3ad-9efe9db46e27" (UID: "a072928c-e8d8-425d-a3ad-9efe9db46e27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:27.255297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.255089 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:27.256622 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.256599 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca" pod "a072928c-e8d8-425d-a3ad-9efe9db46e27" (UID: "a072928c-e8d8-425d-a3ad-9efe9db46e27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:27.256696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.256600 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t" pod "a072928c-e8d8-425d-a3ad-9efe9db46e27" (UID: "a072928c-e8d8-425d-a3ad-9efe9db46e27"). InnerVolumeSpecName "kube-api-access-4r82t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:27.279982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.279959 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-167a01-openshift-default-6b4f5887bf-wsbxk"] Nov 02 03:43:27.356013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.355991 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r82t\" (UniqueName: \"kubernetes.io/projected/a072928c-e8d8-425d-a3ad-9efe9db46e27-kube-api-access-4r82t\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:27.356128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:27.356017 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a072928c-e8d8-425d-a3ad-9efe9db46e27-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:28.074574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:28.074543 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" Nov 02 03:43:28.074574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:28.074564 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l" event={"ID":"a072928c-e8d8-425d-a3ad-9efe9db46e27","Type":"ContainerDied","Data":"2c49fd0ba8efd36cc12475839062dd30acc402e1a1b364612bb344ea13153c43"} Nov 02 03:43:28.075024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:28.074605 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c49fd0ba8efd36cc12475839062dd30acc402e1a1b364612bb344ea13153c43" Nov 02 03:43:28.114443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:28.114421 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" path="/var/lib/kubelet/pods/28f3df67-086d-46a5-85cf-fbc65b88b874/volumes" Nov 02 03:43:33.444313 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:43:33.444279 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:43:38.603380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:38.599294 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d"] Nov 02 03:43:38.754688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:38.754653 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4"] Nov 02 03:43:42.501913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.501880 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf"] Nov 02 03:43:42.502325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502312 4868 cpu_manager.go:414] "containerMap: removing container" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502325 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502338 4868 cpu_manager.go:414] "containerMap: removing container" podUID="106e6e43-0ec0-4353-9212-2596c81cefe3" containerName="revoke-keys" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502344 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106e6e43-0ec0-4353-9212-2596c81cefe3" containerName="revoke-keys" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502372 4868 cpu_manager.go:414] "containerMap: removing container" podUID="90f12570-183f-4c03-9452-1de8eba4be7b" containerName="revoke-keys" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502378 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f12570-183f-4c03-9452-1de8eba4be7b" containerName="revoke-keys" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502387 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a072928c-e8d8-425d-a3ad-9efe9db46e27" containerName="revoke-keys" Nov 02 03:43:42.502398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502391 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a072928c-e8d8-425d-a3ad-9efe9db46e27" containerName="revoke-keys" Nov 02 03:43:42.502614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502473 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a072928c-e8d8-425d-a3ad-9efe9db46e27" containerName="revoke-keys" Nov 02 03:43:42.502614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502484 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" Nov 02 03:43:42.502614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502492 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" Nov 02 03:43:42.502614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502499 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="106e6e43-0ec0-4353-9212-2596c81cefe3" containerName="revoke-keys" Nov 02 03:43:42.502614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.502507 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f12570-183f-4c03-9452-1de8eba4be7b" containerName="revoke-keys" Nov 02 03:43:42.505647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.505627 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.525764 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.525728 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.525875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.525839 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf794\" (UniqueName: \"kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.525926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.525899 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.607472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.607445 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:43:42.607608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.607505 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:43:42.610309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.610287 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf"] Nov 02 03:43:42.626647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.626625 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf794\" (UniqueName: \"kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.626766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.626665 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.626766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.626760 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.627456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.627434 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.629143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.629119 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.683891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.683867 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf794\" (UniqueName: \"kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794\") pod \"maas-api-revoke-sub-afd209fa-nkglf\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:42.818729 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:42.818669 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:43.190343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.190314 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf"] Nov 02 03:43:43.192272 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:43.192248 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7e65f0_4cc4_4a37_9f5f_0e61e7bddd87.slice/crio-66b9223818d744500b9093f8038ebdf28e8abce3ca2ced408def26a084cf814b WatchSource:0}: Error finding container 66b9223818d744500b9093f8038ebdf28e8abce3ca2ced408def26a084cf814b: Status 404 returned error can't find the container with id 66b9223818d744500b9093f8038ebdf28e8abce3ca2ced408def26a084cf814b Nov 02 03:43:43.606324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.606273 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" podUID="aa218932-b8b9-4add-9ecc-304a2516fc22" containerName="payload-processing" containerID="cri-o://682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad" gracePeriod=25 Nov 02 03:43:43.755205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.755152 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" podUID="08dd5c16-1ef7-4965-84ff-441bfb03c477" containerName="payload-pre-processing" containerID="cri-o://42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853" gracePeriod=25 Nov 02 03:43:43.884664 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.884638 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:43:43.940344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.940313 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume\") pod \"aa218932-b8b9-4add-9ecc-304a2516fc22\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " Nov 02 03:43:43.940610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.940476 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g\" (UniqueName: \"kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g\") pod \"aa218932-b8b9-4add-9ecc-304a2516fc22\" (UID: \"aa218932-b8b9-4add-9ecc-304a2516fc22\") " Nov 02 03:43:43.940883 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.940850 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume" pod "aa218932-b8b9-4add-9ecc-304a2516fc22" (UID: "aa218932-b8b9-4add-9ecc-304a2516fc22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:43.943423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.943399 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g" pod "aa218932-b8b9-4add-9ecc-304a2516fc22" (UID: "aa218932-b8b9-4add-9ecc-304a2516fc22"). InnerVolumeSpecName "kube-api-access-dng7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:43.993698 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:43.993677 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:43:44.041310 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.041284 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume\") pod \"08dd5c16-1ef7-4965-84ff-441bfb03c477\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " Nov 02 03:43:44.041444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.041338 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw\" (UniqueName: \"kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw\") pod \"08dd5c16-1ef7-4965-84ff-441bfb03c477\" (UID: \"08dd5c16-1ef7-4965-84ff-441bfb03c477\") " Nov 02 03:43:44.041618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.041605 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa218932-b8b9-4add-9ecc-304a2516fc22-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:44.041658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.041622 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dng7g\" (UniqueName: \"kubernetes.io/projected/aa218932-b8b9-4add-9ecc-304a2516fc22-kube-api-access-dng7g\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:44.041762 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.041732 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume" pod "08dd5c16-1ef7-4965-84ff-441bfb03c477" (UID: "08dd5c16-1ef7-4965-84ff-441bfb03c477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:44.043207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.043187 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw" pod "08dd5c16-1ef7-4965-84ff-441bfb03c477" (UID: "08dd5c16-1ef7-4965-84ff-441bfb03c477"). InnerVolumeSpecName "kube-api-access-mxwcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:44.142716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.142689 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08dd5c16-1ef7-4965-84ff-441bfb03c477-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:44.142716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.142713 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwcw\" (UniqueName: \"kubernetes.io/projected/08dd5c16-1ef7-4965-84ff-441bfb03c477-kube-api-access-mxwcw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:44.190778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.190749 4868 generic.go:356] "Generic (PLEG): container finished" podID="08dd5c16-1ef7-4965-84ff-441bfb03c477" containerID="42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853" exitCode=0 Nov 02 03:43:44.190894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.190795 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" Nov 02 03:43:44.190894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.190830 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" event={"ID":"08dd5c16-1ef7-4965-84ff-441bfb03c477","Type":"ContainerDied","Data":"42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853"} Nov 02 03:43:44.190894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.190884 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4" event={"ID":"08dd5c16-1ef7-4965-84ff-441bfb03c477","Type":"ContainerDied","Data":"8a5a5bcc5b8974fcd34abfc48f5a75f2365a507ca693e5e5e8d7ec739da018f8"} Nov 02 03:43:44.191057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.190915 4868 scope.go:122] "RemoveContainer" containerID="42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853" Nov 02 03:43:44.192489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.192462 4868 generic.go:356] "Generic (PLEG): container finished" podID="bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" containerID="ea76ba97da38616d7f6b3ba7ae67da7eb71e9192e3626b3966cbe29a0bb4cd5f" exitCode=0 Nov 02 03:43:44.192618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.192563 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" event={"ID":"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87","Type":"ContainerDied","Data":"ea76ba97da38616d7f6b3ba7ae67da7eb71e9192e3626b3966cbe29a0bb4cd5f"} Nov 02 03:43:44.192618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.192608 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" event={"ID":"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87","Type":"ContainerStarted","Data":"66b9223818d744500b9093f8038ebdf28e8abce3ca2ced408def26a084cf814b"} Nov 02 03:43:44.194026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.194004 4868 generic.go:356] "Generic (PLEG): container finished" podID="aa218932-b8b9-4add-9ecc-304a2516fc22" containerID="682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad" exitCode=0 Nov 02 03:43:44.194117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.194034 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" event={"ID":"aa218932-b8b9-4add-9ecc-304a2516fc22","Type":"ContainerDied","Data":"682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad"} Nov 02 03:43:44.194117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.194053 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" Nov 02 03:43:44.194117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.194074 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d" event={"ID":"aa218932-b8b9-4add-9ecc-304a2516fc22","Type":"ContainerDied","Data":"86c0717e1b208f16900cd96c06cd10c3582a783ee7bb2e53086c1cfd4f46dc53"} Nov 02 03:43:44.200806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.200783 4868 scope.go:122] "RemoveContainer" containerID="42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853" Nov 02 03:43:44.201065 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:43:44.201043 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853\": container with ID starting with 42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853 not found: ID does not exist" containerID="42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853" Nov 02 03:43:44.201141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.201075 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853"} err="failed to get container status \"42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853\": rpc error: code = NotFound desc = could not find container \"42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853\": container with ID starting with 42076fca3bca21daede08303bd1f754eec0df84e1b160c9650fa99acdb707853 not found: ID does not exist" Nov 02 03:43:44.201141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.201091 4868 scope.go:122] "RemoveContainer" containerID="682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad" Nov 02 03:43:44.209289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.209272 4868 scope.go:122] "RemoveContainer" containerID="682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad" Nov 02 03:43:44.209638 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:43:44.209618 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad\": container with ID starting with 682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad not found: ID does not exist" containerID="682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad" Nov 02 03:43:44.209710 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.209643 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad"} err="failed to get container status \"682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad\": rpc error: code = NotFound desc = could not find container \"682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad\": container with ID starting with 682635b33330c3640d71a4db5f3737039cc12826027fd57d57c17f34a9221aad not found: ID does not exist" Nov 02 03:43:44.354279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.354254 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d"] Nov 02 03:43:44.358316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.358288 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9b06f8-6496f48859-qhv8d"] Nov 02 03:43:44.584779 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.584751 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4"] Nov 02 03:43:44.590057 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:44.590029 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9b06f8-b944bb979-4vxw4"] Nov 02 03:43:45.341267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.341245 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:45.352721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.352693 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794\" (UniqueName: \"kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794\") pod \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " Nov 02 03:43:45.352863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.352844 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca\") pod \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " Nov 02 03:43:45.352995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.352980 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca\") pod \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\" (UID: \"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87\") " Nov 02 03:43:45.353200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.353180 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca" pod "bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" (UID: "bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:45.353329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.353315 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:45.354796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.354771 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794" pod "bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" (UID: "bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87"). InnerVolumeSpecName "kube-api-access-sf794". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:45.354979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.354955 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca" pod "bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" (UID: "bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:45.454379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.454296 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:45.454379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:45.454317 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf794\" (UniqueName: \"kubernetes.io/projected/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87-kube-api-access-sf794\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:46.114499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:46.114458 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08dd5c16-1ef7-4965-84ff-441bfb03c477" path="/var/lib/kubelet/pods/08dd5c16-1ef7-4965-84ff-441bfb03c477/volumes" Nov 02 03:43:46.114911 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:46.114891 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa218932-b8b9-4add-9ecc-304a2516fc22" path="/var/lib/kubelet/pods/aa218932-b8b9-4add-9ecc-304a2516fc22/volumes" Nov 02 03:43:46.209407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:46.209381 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" Nov 02 03:43:46.209534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:46.209400 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf" event={"ID":"bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87","Type":"ContainerDied","Data":"66b9223818d744500b9093f8038ebdf28e8abce3ca2ced408def26a084cf814b"} Nov 02 03:43:46.209534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:46.209454 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66b9223818d744500b9093f8038ebdf28e8abce3ca2ced408def26a084cf814b" Nov 02 03:43:48.677585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.677552 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk"] Nov 02 03:43:48.678003 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.677989 4868 cpu_manager.go:414] "containerMap: removing container" podUID="bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" containerName="revoke-keys" Nov 02 03:43:48.678003 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678003 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" containerName="revoke-keys" Nov 02 03:43:48.678080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678046 4868 cpu_manager.go:414] "containerMap: removing container" podUID="08dd5c16-1ef7-4965-84ff-441bfb03c477" containerName="payload-pre-processing" Nov 02 03:43:48.678080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678052 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08dd5c16-1ef7-4965-84ff-441bfb03c477" containerName="payload-pre-processing" Nov 02 03:43:48.678080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678060 4868 cpu_manager.go:414] "containerMap: removing container" podUID="aa218932-b8b9-4add-9ecc-304a2516fc22" containerName="payload-processing" Nov 02 03:43:48.678080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678064 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa218932-b8b9-4add-9ecc-304a2516fc22" containerName="payload-processing" Nov 02 03:43:48.678080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678075 4868 cpu_manager.go:414] "containerMap: removing container" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" Nov 02 03:43:48.678080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678079 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f3df67-086d-46a5-85cf-fbc65b88b874" containerName="istio-proxy" Nov 02 03:43:48.678266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678168 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" containerName="revoke-keys" Nov 02 03:43:48.678266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678177 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="08dd5c16-1ef7-4965-84ff-441bfb03c477" containerName="payload-pre-processing" Nov 02 03:43:48.678266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.678184 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa218932-b8b9-4add-9ecc-304a2516fc22" containerName="payload-processing" Nov 02 03:43:48.681445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.681430 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.737698 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.737667 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk"] Nov 02 03:43:48.783116 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.783090 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:43:48.783255 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.783157 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:43:48.786664 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.786640 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.786819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.786801 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5glrr\" (UniqueName: \"kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.786882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.786854 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.887502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.887432 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5glrr\" (UniqueName: \"kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.887502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.887477 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.887686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.887522 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.888121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.888101 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.889813 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.889792 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.943889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.943822 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5glrr\" (UniqueName: \"kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr\") pod \"maas-api-revoke-sub-521a73dc-nzsbk\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:48.995457 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:48.995434 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:49.363770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:49.363742 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk"] Nov 02 03:43:49.365171 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:43:49.365139 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf38542e_ce0d_46e6_8511_34dc80dce119.slice/crio-e8dab10372dabb04a4d320321548f2800af558a6524541e3b57c4d1804344d62 WatchSource:0}: Error finding container e8dab10372dabb04a4d320321548f2800af558a6524541e3b57c4d1804344d62: Status 404 returned error can't find the container with id e8dab10372dabb04a4d320321548f2800af558a6524541e3b57c4d1804344d62 Nov 02 03:43:50.238022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:50.237990 4868 generic.go:356] "Generic (PLEG): container finished" podID="cf38542e-ce0d-46e6-8511-34dc80dce119" containerID="e2e04f14acf8905d38e1cdd948ab8d209c3549136db6a5973e4ecd235a811551" exitCode=0 Nov 02 03:43:50.238427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:50.238053 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" event={"ID":"cf38542e-ce0d-46e6-8511-34dc80dce119","Type":"ContainerDied","Data":"e2e04f14acf8905d38e1cdd948ab8d209c3549136db6a5973e4ecd235a811551"} Nov 02 03:43:50.238427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:50.238107 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" event={"ID":"cf38542e-ce0d-46e6-8511-34dc80dce119","Type":"ContainerStarted","Data":"e8dab10372dabb04a4d320321548f2800af558a6524541e3b57c4d1804344d62"} Nov 02 03:43:51.380198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.380167 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:51.419514 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.419481 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca\") pod \"cf38542e-ce0d-46e6-8511-34dc80dce119\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " Nov 02 03:43:51.419689 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.419646 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca\") pod \"cf38542e-ce0d-46e6-8511-34dc80dce119\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " Nov 02 03:43:51.419757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.419693 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr\" (UniqueName: \"kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr\") pod \"cf38542e-ce0d-46e6-8511-34dc80dce119\" (UID: \"cf38542e-ce0d-46e6-8511-34dc80dce119\") " Nov 02 03:43:51.419872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.419845 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca" pod "cf38542e-ce0d-46e6-8511-34dc80dce119" (UID: "cf38542e-ce0d-46e6-8511-34dc80dce119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:43:51.420036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.420019 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:51.421693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.421672 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca" pod "cf38542e-ce0d-46e6-8511-34dc80dce119" (UID: "cf38542e-ce0d-46e6-8511-34dc80dce119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:43:51.421789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.421765 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr" pod "cf38542e-ce0d-46e6-8511-34dc80dce119" (UID: "cf38542e-ce0d-46e6-8511-34dc80dce119"). InnerVolumeSpecName "kube-api-access-5glrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:43:51.527434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.522726 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf38542e-ce0d-46e6-8511-34dc80dce119-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:51.527434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:51.522769 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5glrr\" (UniqueName: \"kubernetes.io/projected/cf38542e-ce0d-46e6-8511-34dc80dce119-kube-api-access-5glrr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:43:52.252412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:52.252373 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" event={"ID":"cf38542e-ce0d-46e6-8511-34dc80dce119","Type":"ContainerDied","Data":"e8dab10372dabb04a4d320321548f2800af558a6524541e3b57c4d1804344d62"} Nov 02 03:43:52.252412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:52.252397 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk" Nov 02 03:43:52.252679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:52.252424 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8dab10372dabb04a4d320321548f2800af558a6524541e3b57c4d1804344d62" Nov 02 03:43:55.431078 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:55.431046 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th"] Nov 02 03:43:55.480470 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:55.480441 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-2912e6-wb4th"] Nov 02 03:43:56.114712 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:43:56.114680 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fb0115-6e81-4075-ac5a-91db7c60a68c" path="/var/lib/kubelet/pods/b2fb0115-6e81-4075-ac5a-91db7c60a68c/volumes" Nov 02 03:44:01.639498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.639461 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2"] Nov 02 03:44:01.640102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.640084 4868 cpu_manager.go:414] "containerMap: removing container" podUID="cf38542e-ce0d-46e6-8511-34dc80dce119" containerName="revoke-keys" Nov 02 03:44:01.640152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.640103 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf38542e-ce0d-46e6-8511-34dc80dce119" containerName="revoke-keys" Nov 02 03:44:01.640225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.640215 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf38542e-ce0d-46e6-8511-34dc80dce119" containerName="revoke-keys" Nov 02 03:44:01.644725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.644710 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.696444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.696407 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-858236-openshift-default-dockercfg-79wxp\"" Nov 02 03:44:01.740947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.740897 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.740952 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741011 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741062 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741148 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741187 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq95h\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741217 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741441 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.741482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.741467 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.749976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.749948 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2"] Nov 02 03:44:01.842462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842430 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842464 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842538 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842556 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842583 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842628 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842658 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.842718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842691 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq95h\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.843041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.842719 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.843041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.843021 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.843324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.843296 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.843444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.843326 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.843444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.843327 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.843444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.843390 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.845193 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.845167 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.845316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.845296 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.898787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.898716 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.898926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.898835 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq95h\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h\") pod \"e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:01.957486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:01.957445 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:02.393142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:02.393101 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2"] Nov 02 03:44:02.403752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:02.403713 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:44:02.403880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:02.403788 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:44:02.403880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:02.403818 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:44:03.331370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:03.331305 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" event={"ID":"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2","Type":"ContainerStarted","Data":"3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1"} Nov 02 03:44:03.331894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:03.331409 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" event={"ID":"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2","Type":"ContainerStarted","Data":"4dd445c08da92394bb82e38d6b6ab6c2cc9e98549b19611ed3d3510ca1982e80"} Nov 02 03:44:03.467610 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:44:03.467569 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:44:03.556471 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:03.548766 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podStartSLOduration=2.548747244 podStartE2EDuration="2.548747244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:44:03.495515532 +0000 UTC m=+3385.491216476" watchObservedRunningTime="2026-11-02 03:44:03.548747244 +0000 UTC m=+3385.544448188" Nov 02 03:44:03.957645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:03.957607 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:03.959522 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:03.959493 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 03:44:03.959660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:03.959543 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 03:44:04.958063 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:04.958022 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 03:44:04.958482 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:04.958080 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 03:44:05.958771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:05.958734 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 03:44:05.959184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:05.958786 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 03:44:06.958060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:06.958021 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 03:44:06.958256 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:06.958087 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 03:44:07.499115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.499079 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w"] Nov 02 03:44:07.502812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.502796 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.559340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.559303 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:44:07.560686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.560662 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w"] Nov 02 03:44:07.606154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.606127 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:44:07.607615 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.607593 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.607674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.607630 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.607831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.607807 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqf56\" (UniqueName: \"kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.629719 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.629690 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s"] Nov 02 03:44:07.657658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.657624 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.690474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.690439 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s"] Nov 02 03:44:07.708782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.708750 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.708910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.708800 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.709455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.709431 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqf56\" (UniqueName: \"kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.709524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.709470 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.711505 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.711457 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.766089 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.766005 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqf56\" (UniqueName: \"kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.810970 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.810941 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.811158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.810979 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcc82\" (UniqueName: \"kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.811158 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.811002 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.816687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.816664 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:07.912307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.911997 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.912307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.912198 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.912307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.912237 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcc82\" (UniqueName: \"kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.915147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.915100 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.915686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.915656 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.959117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.958697 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 03:44:07.959117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.958749 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 03:44:07.968741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.968714 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcc82\" (UniqueName: \"kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82\") pod \"maas-api-revoke-sub-3ede3cdd-sb89s\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:07.974796 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:07.974772 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:08.217605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:08.217580 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w"] Nov 02 03:44:08.218023 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:08.218004 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fd83e8_8343_4b93_986f_62ce0aebd375.slice/crio-7ffed7194c17a2c4ad0b504400066f160bd2a22a1625fe5a60b556433f17c5e2 WatchSource:0}: Error finding container 7ffed7194c17a2c4ad0b504400066f160bd2a22a1625fe5a60b556433f17c5e2: Status 404 returned error can't find the container with id 7ffed7194c17a2c4ad0b504400066f160bd2a22a1625fe5a60b556433f17c5e2 Nov 02 03:44:08.359824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:08.359778 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s"] Nov 02 03:44:08.362384 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:08.362333 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc02105_b67e_4857_9ed9_6610af43d378.slice/crio-c7d02ef9d9180b0977a40204e1a10fd32f7e291d9e3a5a030cf0be959f5a566f WatchSource:0}: Error finding container c7d02ef9d9180b0977a40204e1a10fd32f7e291d9e3a5a030cf0be959f5a566f: Status 404 returned error can't find the container with id c7d02ef9d9180b0977a40204e1a10fd32f7e291d9e3a5a030cf0be959f5a566f Nov 02 03:44:08.385212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:08.385188 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" event={"ID":"76fd83e8-8343-4b93-986f-62ce0aebd375","Type":"ContainerStarted","Data":"7ffed7194c17a2c4ad0b504400066f160bd2a22a1625fe5a60b556433f17c5e2"} Nov 02 03:44:08.386343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:08.386322 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" event={"ID":"cbc02105-b67e-4857-9ed9-6610af43d378","Type":"ContainerStarted","Data":"c7d02ef9d9180b0977a40204e1a10fd32f7e291d9e3a5a030cf0be959f5a566f"} Nov 02 03:44:08.958974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:08.958941 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 03:44:08.959437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:08.958995 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 03:44:09.396129 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.396098 4868 generic.go:356] "Generic (PLEG): container finished" podID="76fd83e8-8343-4b93-986f-62ce0aebd375" containerID="191d93bc14500b193f8140429547df3dd0899da4e6a3ec08a40632fa0bb8a7e0" exitCode=0 Nov 02 03:44:09.396276 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.396134 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" event={"ID":"76fd83e8-8343-4b93-986f-62ce0aebd375","Type":"ContainerDied","Data":"191d93bc14500b193f8140429547df3dd0899da4e6a3ec08a40632fa0bb8a7e0"} Nov 02 03:44:09.398126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.398095 4868 generic.go:356] "Generic (PLEG): container finished" podID="cbc02105-b67e-4857-9ed9-6610af43d378" containerID="f9a125ead5344a377ff87cd690bc4d1c5c608b732563bba6d87bc9604c1ba696" exitCode=0 Nov 02 03:44:09.398226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.398174 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" event={"ID":"cbc02105-b67e-4857-9ed9-6610af43d378","Type":"ContainerDied","Data":"f9a125ead5344a377ff87cd690bc4d1c5c608b732563bba6d87bc9604c1ba696"} Nov 02 03:44:09.962197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.962165 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:09.962715 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.962449 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:09.963156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:09.963137 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:44:10.571673 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.571645 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:10.577608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.577587 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:10.639047 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639017 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca\") pod \"76fd83e8-8343-4b93-986f-62ce0aebd375\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " Nov 02 03:44:10.639190 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639067 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca\") pod \"76fd83e8-8343-4b93-986f-62ce0aebd375\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " Nov 02 03:44:10.639190 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639096 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82\" (UniqueName: \"kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82\") pod \"cbc02105-b67e-4857-9ed9-6610af43d378\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " Nov 02 03:44:10.639190 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639129 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca\") pod \"cbc02105-b67e-4857-9ed9-6610af43d378\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " Nov 02 03:44:10.639325 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639290 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56\" (UniqueName: \"kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56\") pod \"76fd83e8-8343-4b93-986f-62ce0aebd375\" (UID: \"76fd83e8-8343-4b93-986f-62ce0aebd375\") " Nov 02 03:44:10.639400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639347 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca\") pod \"cbc02105-b67e-4857-9ed9-6610af43d378\" (UID: \"cbc02105-b67e-4857-9ed9-6610af43d378\") " Nov 02 03:44:10.639522 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639492 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca" pod "76fd83e8-8343-4b93-986f-62ce0aebd375" (UID: "76fd83e8-8343-4b93-986f-62ce0aebd375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:10.639626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639601 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca" pod "cbc02105-b67e-4857-9ed9-6610af43d378" (UID: "cbc02105-b67e-4857-9ed9-6610af43d378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:10.639778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639757 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:10.639853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.639783 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:10.641217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.641197 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca" pod "76fd83e8-8343-4b93-986f-62ce0aebd375" (UID: "76fd83e8-8343-4b93-986f-62ce0aebd375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:10.641481 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.641449 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56" pod "76fd83e8-8343-4b93-986f-62ce0aebd375" (UID: "76fd83e8-8343-4b93-986f-62ce0aebd375"). InnerVolumeSpecName "kube-api-access-lqf56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:10.641586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.641565 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82" pod "cbc02105-b67e-4857-9ed9-6610af43d378" (UID: "cbc02105-b67e-4857-9ed9-6610af43d378"). InnerVolumeSpecName "kube-api-access-wcc82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:10.641786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.641761 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca" pod "cbc02105-b67e-4857-9ed9-6610af43d378" (UID: "cbc02105-b67e-4857-9ed9-6610af43d378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:10.741212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.741137 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76fd83e8-8343-4b93-986f-62ce0aebd375-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:10.741212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.741167 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcc82\" (UniqueName: \"kubernetes.io/projected/cbc02105-b67e-4857-9ed9-6610af43d378-kube-api-access-wcc82\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:10.741212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.741184 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqf56\" (UniqueName: \"kubernetes.io/projected/76fd83e8-8343-4b93-986f-62ce0aebd375-kube-api-access-lqf56\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:10.741212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:10.741197 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc02105-b67e-4857-9ed9-6610af43d378-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:11.413430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:11.413397 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" Nov 02 03:44:11.413430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:11.413422 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w" event={"ID":"76fd83e8-8343-4b93-986f-62ce0aebd375","Type":"ContainerDied","Data":"7ffed7194c17a2c4ad0b504400066f160bd2a22a1625fe5a60b556433f17c5e2"} Nov 02 03:44:11.414003 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:11.413475 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ffed7194c17a2c4ad0b504400066f160bd2a22a1625fe5a60b556433f17c5e2" Nov 02 03:44:11.414898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:11.414876 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" Nov 02 03:44:11.415005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:11.414902 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s" event={"ID":"cbc02105-b67e-4857-9ed9-6610af43d378","Type":"ContainerDied","Data":"c7d02ef9d9180b0977a40204e1a10fd32f7e291d9e3a5a030cf0be959f5a566f"} Nov 02 03:44:11.415005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:11.414938 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d02ef9d9180b0977a40204e1a10fd32f7e291d9e3a5a030cf0be959f5a566f" Nov 02 03:44:22.998997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.998962 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj"] Nov 02 03:44:22.999657 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.999637 4868 cpu_manager.go:414] "containerMap: removing container" podUID="76fd83e8-8343-4b93-986f-62ce0aebd375" containerName="revoke-keys" Nov 02 03:44:22.999753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.999659 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fd83e8-8343-4b93-986f-62ce0aebd375" containerName="revoke-keys" Nov 02 03:44:22.999753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.999686 4868 cpu_manager.go:414] "containerMap: removing container" podUID="cbc02105-b67e-4857-9ed9-6610af43d378" containerName="revoke-keys" Nov 02 03:44:22.999753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.999695 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc02105-b67e-4857-9ed9-6610af43d378" containerName="revoke-keys" Nov 02 03:44:22.999909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.999824 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc02105-b67e-4857-9ed9-6610af43d378" containerName="revoke-keys" Nov 02 03:44:22.999909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:22.999846 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fd83e8-8343-4b93-986f-62ce0aebd375" containerName="revoke-keys" Nov 02 03:44:23.004404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.004382 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.060754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.060727 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj"] Nov 02 03:44:23.105606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.105577 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:44:23.105606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.105595 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:44:23.181242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.181214 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpdv\" (UniqueName: \"kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.181389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.181363 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.181462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.181439 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.282484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.282412 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpdv\" (UniqueName: \"kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.282613 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.282541 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.282613 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.282578 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.283264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.283236 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.284702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.284678 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.338340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.338318 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpdv\" (UniqueName: \"kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv\") pod \"maas-api-revoke-sub-fbcea670-cnlrj\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.618757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.618731 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:23.994108 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:23.994079 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj"] Nov 02 03:44:23.994228 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:23.994188 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d94b01f_307b_4d89_987c_18cadad403c7.slice/crio-f0cb60d0a18287541bfd54eab44a8edfb94bcea5619902331c8c084e798bb358 WatchSource:0}: Error finding container f0cb60d0a18287541bfd54eab44a8edfb94bcea5619902331c8c084e798bb358: Status 404 returned error can't find the container with id f0cb60d0a18287541bfd54eab44a8edfb94bcea5619902331c8c084e798bb358 Nov 02 03:44:24.514400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:24.501281 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" event={"ID":"2d94b01f-307b-4d89-987c-18cadad403c7","Type":"ContainerStarted","Data":"798ea1e204b199f47aa2231dba9ec26291a6bb690678e4d453ce0070428effc8"} Nov 02 03:44:24.514400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:24.501365 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" event={"ID":"2d94b01f-307b-4d89-987c-18cadad403c7","Type":"ContainerStarted","Data":"f0cb60d0a18287541bfd54eab44a8edfb94bcea5619902331c8c084e798bb358"} Nov 02 03:44:24.627374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:24.626450 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" podStartSLOduration=2.62643025 podStartE2EDuration="2.62643025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:44:24.614468051 +0000 UTC m=+3406.610168996" watchObservedRunningTime="2026-11-02 03:44:24.62643025 +0000 UTC m=+3406.622131195" Nov 02 03:44:25.531396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:25.530867 4868 generic.go:356] "Generic (PLEG): container finished" podID="2d94b01f-307b-4d89-987c-18cadad403c7" containerID="798ea1e204b199f47aa2231dba9ec26291a6bb690678e4d453ce0070428effc8" exitCode=0 Nov 02 03:44:25.531396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:25.531037 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" event={"ID":"2d94b01f-307b-4d89-987c-18cadad403c7","Type":"ContainerDied","Data":"798ea1e204b199f47aa2231dba9ec26291a6bb690678e4d453ce0070428effc8"} Nov 02 03:44:26.685310 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.685281 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:26.779787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.779761 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv\" (UniqueName: \"kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv\") pod \"2d94b01f-307b-4d89-987c-18cadad403c7\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " Nov 02 03:44:26.779918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.779868 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca\") pod \"2d94b01f-307b-4d89-987c-18cadad403c7\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " Nov 02 03:44:26.779968 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.779929 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca\") pod \"2d94b01f-307b-4d89-987c-18cadad403c7\" (UID: \"2d94b01f-307b-4d89-987c-18cadad403c7\") " Nov 02 03:44:26.780347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.780322 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca" pod "2d94b01f-307b-4d89-987c-18cadad403c7" (UID: "2d94b01f-307b-4d89-987c-18cadad403c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:26.781814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.781784 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca" pod "2d94b01f-307b-4d89-987c-18cadad403c7" (UID: "2d94b01f-307b-4d89-987c-18cadad403c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:26.781905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.781847 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv" pod "2d94b01f-307b-4d89-987c-18cadad403c7" (UID: "2d94b01f-307b-4d89-987c-18cadad403c7"). InnerVolumeSpecName "kube-api-access-wnpdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:26.881596 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.881559 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:26.881596 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.881596 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d94b01f-307b-4d89-987c-18cadad403c7-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:26.881797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:26.881613 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpdv\" (UniqueName: \"kubernetes.io/projected/2d94b01f-307b-4d89-987c-18cadad403c7-kube-api-access-wnpdv\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:27.126809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:27.126770 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj"] Nov 02 03:44:27.129827 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:27.129796 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9b06f8-ld9xj"] Nov 02 03:44:27.544687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:27.544607 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" event={"ID":"2d94b01f-307b-4d89-987c-18cadad403c7","Type":"ContainerDied","Data":"f0cb60d0a18287541bfd54eab44a8edfb94bcea5619902331c8c084e798bb358"} Nov 02 03:44:27.544687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:27.544636 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj" Nov 02 03:44:27.544687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:27.544663 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0cb60d0a18287541bfd54eab44a8edfb94bcea5619902331c8c084e798bb358" Nov 02 03:44:28.114805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:28.114770 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f12570-183f-4c03-9452-1de8eba4be7b" path="/var/lib/kubelet/pods/90f12570-183f-4c03-9452-1de8eba4be7b/volumes" Nov 02 03:44:29.066068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.066029 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg"] Nov 02 03:44:29.066578 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.066562 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2d94b01f-307b-4d89-987c-18cadad403c7" containerName="revoke-keys" Nov 02 03:44:29.066634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.066579 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d94b01f-307b-4d89-987c-18cadad403c7" containerName="revoke-keys" Nov 02 03:44:29.066677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.066666 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d94b01f-307b-4d89-987c-18cadad403c7" containerName="revoke-keys" Nov 02 03:44:29.070212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.070196 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.100850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.100823 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.101008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.100863 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc48j\" (UniqueName: \"kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.101008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.100885 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.126638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.126611 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg"] Nov 02 03:44:29.172185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.172154 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:44:29.172320 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.172201 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:44:29.201804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.201778 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.201914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.201820 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc48j\" (UniqueName: \"kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.202005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.201986 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.202490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.202470 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.204272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.204250 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.257983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.257950 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc48j\" (UniqueName: \"kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j\") pod \"maas-api-revoke-sub-db3591af-j2vfg\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.383324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.383290 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:29.754973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:29.754943 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg"] Nov 02 03:44:29.755935 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:29.755908 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab7ecb8_8f2b_4654_82d8_86dcd1d1d1a6.slice/crio-69f95c5e43c97c1a608ca3f45ef27b54ce2240032ff43e91c2b6787880341fe3 WatchSource:0}: Error finding container 69f95c5e43c97c1a608ca3f45ef27b54ce2240032ff43e91c2b6787880341fe3: Status 404 returned error can't find the container with id 69f95c5e43c97c1a608ca3f45ef27b54ce2240032ff43e91c2b6787880341fe3 Nov 02 03:44:30.564794 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:30.564764 4868 generic.go:356] "Generic (PLEG): container finished" podID="6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" containerID="0c5555a23ca3ec4b6ba1eb9403bf5de44219a93ce2b555753e3af82985b3cb7e" exitCode=0 Nov 02 03:44:30.565104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:30.564837 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" event={"ID":"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6","Type":"ContainerDied","Data":"0c5555a23ca3ec4b6ba1eb9403bf5de44219a93ce2b555753e3af82985b3cb7e"} Nov 02 03:44:30.565104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:30.564877 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" event={"ID":"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6","Type":"ContainerStarted","Data":"69f95c5e43c97c1a608ca3f45ef27b54ce2240032ff43e91c2b6787880341fe3"} Nov 02 03:44:31.720213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.720184 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:31.725961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.725942 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca\") pod \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " Nov 02 03:44:31.726037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.725980 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca\") pod \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " Nov 02 03:44:31.726037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.726014 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j\" (UniqueName: \"kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j\") pod \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\" (UID: \"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6\") " Nov 02 03:44:31.726334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.726306 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca" pod "6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" (UID: "6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:31.728021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.727999 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j" pod "6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" (UID: "6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6"). InnerVolumeSpecName "kube-api-access-pc48j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:31.728117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.728027 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca" pod "6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" (UID: "6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:31.827540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.827486 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:31.827540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.827509 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:31.827540 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:31.827518 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc48j\" (UniqueName: \"kubernetes.io/projected/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6-kube-api-access-pc48j\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:32.578701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:32.578619 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" Nov 02 03:44:32.578701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:32.578662 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg" event={"ID":"6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6","Type":"ContainerDied","Data":"69f95c5e43c97c1a608ca3f45ef27b54ce2240032ff43e91c2b6787880341fe3"} Nov 02 03:44:32.578944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:32.578722 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f95c5e43c97c1a608ca3f45ef27b54ce2240032ff43e91c2b6787880341fe3" Nov 02 03:44:33.463175 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:44:33.463134 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:44:38.847336 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.847300 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j"] Nov 02 03:44:38.847945 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.847924 4868 cpu_manager.go:414] "containerMap: removing container" podUID="6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" containerName="revoke-keys" Nov 02 03:44:38.848029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.847947 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" containerName="revoke-keys" Nov 02 03:44:38.848147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.848133 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" containerName="revoke-keys" Nov 02 03:44:38.852518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.852497 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:38.901289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.901258 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv65x\" (UniqueName: \"kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:38.901404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.901317 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:38.901404 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.901389 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:38.907546 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.907526 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j"] Nov 02 03:44:38.952811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.952786 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:44:38.952914 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:38.952834 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:44:39.002517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.002492 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv65x\" (UniqueName: \"kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.002623 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.002580 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.002623 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.002607 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.003194 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.003172 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.004778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.004758 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.059103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.059079 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv65x\" (UniqueName: \"kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x\") pod \"maas-api-revoke-sub-bb1515ee-mwk9j\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.165146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.165117 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:39.536475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.536445 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j"] Nov 02 03:44:39.537030 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:39.537005 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4732e0c_3043_4985_ab02_78bb7b7060b2.slice/crio-6da1312e92c8dbebab5a0fc0e89aaefe6c2d59c9210c04024c9f9197d2378cc0 WatchSource:0}: Error finding container 6da1312e92c8dbebab5a0fc0e89aaefe6c2d59c9210c04024c9f9197d2378cc0: Status 404 returned error can't find the container with id 6da1312e92c8dbebab5a0fc0e89aaefe6c2d59c9210c04024c9f9197d2378cc0 Nov 02 03:44:39.649655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:39.649619 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" event={"ID":"b4732e0c-3043-4985-ab02-78bb7b7060b2","Type":"ContainerStarted","Data":"6da1312e92c8dbebab5a0fc0e89aaefe6c2d59c9210c04024c9f9197d2378cc0"} Nov 02 03:44:40.657381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:40.657326 4868 generic.go:356] "Generic (PLEG): container finished" podID="b4732e0c-3043-4985-ab02-78bb7b7060b2" containerID="46d33280db72e44c13156a33287dff50f499bd53f480053cfb1c2787c5704720" exitCode=0 Nov 02 03:44:40.657821 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:40.657409 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" event={"ID":"b4732e0c-3043-4985-ab02-78bb7b7060b2","Type":"ContainerDied","Data":"46d33280db72e44c13156a33287dff50f499bd53f480053cfb1c2787c5704720"} Nov 02 03:44:41.805832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.805805 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:41.931280 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.931210 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca\") pod \"b4732e0c-3043-4985-ab02-78bb7b7060b2\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " Nov 02 03:44:41.931280 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.931263 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca\") pod \"b4732e0c-3043-4985-ab02-78bb7b7060b2\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " Nov 02 03:44:41.931517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.931303 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x\" (UniqueName: \"kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x\") pod \"b4732e0c-3043-4985-ab02-78bb7b7060b2\" (UID: \"b4732e0c-3043-4985-ab02-78bb7b7060b2\") " Nov 02 03:44:41.931637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.931612 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca" pod "b4732e0c-3043-4985-ab02-78bb7b7060b2" (UID: "b4732e0c-3043-4985-ab02-78bb7b7060b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:41.931789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.931773 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:41.933323 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.933299 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca" pod "b4732e0c-3043-4985-ab02-78bb7b7060b2" (UID: "b4732e0c-3043-4985-ab02-78bb7b7060b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:41.933865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:41.933838 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x" pod "b4732e0c-3043-4985-ab02-78bb7b7060b2" (UID: "b4732e0c-3043-4985-ab02-78bb7b7060b2"). InnerVolumeSpecName "kube-api-access-fv65x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:42.032549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:42.032524 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4732e0c-3043-4985-ab02-78bb7b7060b2-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:42.032549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:42.032547 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv65x\" (UniqueName: \"kubernetes.io/projected/b4732e0c-3043-4985-ab02-78bb7b7060b2-kube-api-access-fv65x\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:42.671642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:42.671608 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" Nov 02 03:44:42.671811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:42.671640 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j" event={"ID":"b4732e0c-3043-4985-ab02-78bb7b7060b2","Type":"ContainerDied","Data":"6da1312e92c8dbebab5a0fc0e89aaefe6c2d59c9210c04024c9f9197d2378cc0"} Nov 02 03:44:42.671811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:42.671693 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da1312e92c8dbebab5a0fc0e89aaefe6c2d59c9210c04024c9f9197d2378cc0" Nov 02 03:44:44.543270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.543237 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7"] Nov 02 03:44:44.543686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.543671 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b4732e0c-3043-4985-ab02-78bb7b7060b2" containerName="revoke-keys" Nov 02 03:44:44.543686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.543682 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4732e0c-3043-4985-ab02-78bb7b7060b2" containerName="revoke-keys" Nov 02 03:44:44.543793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.543783 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4732e0c-3043-4985-ab02-78bb7b7060b2" containerName="revoke-keys" Nov 02 03:44:44.547184 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.547167 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.603370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.603317 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7"] Nov 02 03:44:44.648450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.648422 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:44:44.648593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.648496 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:44:44.658183 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.658153 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.658306 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.658219 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.658417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.658385 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gxz\" (UniqueName: \"kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.771375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.766367 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.771375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.766535 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gxz\" (UniqueName: \"kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.771375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.766632 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.787139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.787104 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.790531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.790503 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.833693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.833611 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gxz\" (UniqueName: \"kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz\") pod \"maas-api-revoke-sub-e18790f4-fslq7\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:44.860419 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:44.860384 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:45.276299 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:45.276263 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7"] Nov 02 03:44:45.277126 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:45.277092 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4e90fd_1ab5_41c6_bfbc_e23629a7167b.slice/crio-8c67197aa80a82d6b4a75089084b622d23436fa640df0e0319958571b56a60e2 WatchSource:0}: Error finding container 8c67197aa80a82d6b4a75089084b622d23436fa640df0e0319958571b56a60e2: Status 404 returned error can't find the container with id 8c67197aa80a82d6b4a75089084b622d23436fa640df0e0319958571b56a60e2 Nov 02 03:44:45.699090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:45.699054 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" event={"ID":"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b","Type":"ContainerStarted","Data":"8c67197aa80a82d6b4a75089084b622d23436fa640df0e0319958571b56a60e2"} Nov 02 03:44:46.541481 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.541407 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s"] Nov 02 03:44:46.545240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.545222 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.597907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.597830 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrdq4\" (UniqueName: \"kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.598054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.597984 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.598119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.598080 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.601393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.601368 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s"] Nov 02 03:44:46.699450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.699415 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrdq4\" (UniqueName: \"kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.699864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.699486 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.699864 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.699570 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.700142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.700115 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.701972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.701951 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.707642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.707615 4868 generic.go:356] "Generic (PLEG): container finished" podID="9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" containerID="4a55e18a89aabd15d9f77922b99dc853a3c7d1634a1b58c8f76f23a52dbe0207" exitCode=0 Nov 02 03:44:46.707744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.707650 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" event={"ID":"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b","Type":"ContainerDied","Data":"4a55e18a89aabd15d9f77922b99dc853a3c7d1634a1b58c8f76f23a52dbe0207"} Nov 02 03:44:46.756216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.756196 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrdq4\" (UniqueName: \"kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4\") pod \"maas-api-revoke-sub-e870276d-6m95s\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:46.858676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:46.858648 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:47.234328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.234273 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s"] Nov 02 03:44:47.235317 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:44:47.235260 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ecae1e_6d09_48d9_b01b_2d18898e5364.slice/crio-675c764ccb7079c441aef6045166b4b47cc19c17378bb2d089c38135a9810303 WatchSource:0}: Error finding container 675c764ccb7079c441aef6045166b4b47cc19c17378bb2d089c38135a9810303: Status 404 returned error can't find the container with id 675c764ccb7079c441aef6045166b4b47cc19c17378bb2d089c38135a9810303 Nov 02 03:44:47.716319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.716284 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" event={"ID":"37ecae1e-6d09-48d9-b01b-2d18898e5364","Type":"ContainerStarted","Data":"01cea48d88bdb2cbadc9425b57e6541d5c68b5c05cc97b1aa664c39abb09add4"} Nov 02 03:44:47.716703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.716340 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" event={"ID":"37ecae1e-6d09-48d9-b01b-2d18898e5364","Type":"ContainerStarted","Data":"675c764ccb7079c441aef6045166b4b47cc19c17378bb2d089c38135a9810303"} Nov 02 03:44:47.832553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.832459 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" podStartSLOduration=1.832443308 podStartE2EDuration="1.832443308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:44:47.828677259 +0000 UTC m=+3429.824378206" watchObservedRunningTime="2026-11-02 03:44:47.832443308 +0000 UTC m=+3429.828144250" Nov 02 03:44:47.851031 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.851008 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:47.911859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.911830 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz\" (UniqueName: \"kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz\") pod \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " Nov 02 03:44:47.911991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.911874 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca\") pod \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " Nov 02 03:44:47.911991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.911934 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca\") pod \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\" (UID: \"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b\") " Nov 02 03:44:47.912221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.912197 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca" pod "9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" (UID: "9b4e90fd-1ab5-41c6-bfbc-e23629a7167b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:47.912423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.912402 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:47.913947 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.913919 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca" pod "9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" (UID: "9b4e90fd-1ab5-41c6-bfbc-e23629a7167b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:47.914040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:47.913997 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz" pod "9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" (UID: "9b4e90fd-1ab5-41c6-bfbc-e23629a7167b"). InnerVolumeSpecName "kube-api-access-l9gxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:48.013289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.013263 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gxz\" (UniqueName: \"kubernetes.io/projected/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-kube-api-access-l9gxz\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:48.013289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.013289 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:48.724165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.724134 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:44:48.724602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.724167 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" event={"ID":"9b4e90fd-1ab5-41c6-bfbc-e23629a7167b","Type":"ContainerDied","Data":"8c67197aa80a82d6b4a75089084b622d23436fa640df0e0319958571b56a60e2"} Nov 02 03:44:48.724602 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.724214 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c67197aa80a82d6b4a75089084b622d23436fa640df0e0319958571b56a60e2" Nov 02 03:44:48.725865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.725838 4868 generic.go:356] "Generic (PLEG): container finished" podID="37ecae1e-6d09-48d9-b01b-2d18898e5364" containerID="01cea48d88bdb2cbadc9425b57e6541d5c68b5c05cc97b1aa664c39abb09add4" exitCode=0 Nov 02 03:44:48.725992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:48.725932 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" event={"ID":"37ecae1e-6d09-48d9-b01b-2d18898e5364","Type":"ContainerDied","Data":"01cea48d88bdb2cbadc9425b57e6541d5c68b5c05cc97b1aa664c39abb09add4"} Nov 02 03:44:49.870912 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.870889 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:49.932773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.932746 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca\") pod \"37ecae1e-6d09-48d9-b01b-2d18898e5364\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " Nov 02 03:44:49.932894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.932817 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4\" (UniqueName: \"kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4\") pod \"37ecae1e-6d09-48d9-b01b-2d18898e5364\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " Nov 02 03:44:49.932894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.932874 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca\") pod \"37ecae1e-6d09-48d9-b01b-2d18898e5364\" (UID: \"37ecae1e-6d09-48d9-b01b-2d18898e5364\") " Nov 02 03:44:49.933115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.933096 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca" pod "37ecae1e-6d09-48d9-b01b-2d18898e5364" (UID: "37ecae1e-6d09-48d9-b01b-2d18898e5364"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:44:49.933207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.933193 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:49.934986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.934961 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4" pod "37ecae1e-6d09-48d9-b01b-2d18898e5364" (UID: "37ecae1e-6d09-48d9-b01b-2d18898e5364"). InnerVolumeSpecName "kube-api-access-wrdq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:44:49.935270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:49.935239 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca" pod "37ecae1e-6d09-48d9-b01b-2d18898e5364" (UID: "37ecae1e-6d09-48d9-b01b-2d18898e5364"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:44:50.034088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:50.034021 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ecae1e-6d09-48d9-b01b-2d18898e5364-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:50.034088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:50.034045 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrdq4\" (UniqueName: \"kubernetes.io/projected/37ecae1e-6d09-48d9-b01b-2d18898e5364-kube-api-access-wrdq4\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:44:50.738250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:50.738223 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" Nov 02 03:44:50.738440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:50.738265 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s" event={"ID":"37ecae1e-6d09-48d9-b01b-2d18898e5364","Type":"ContainerDied","Data":"675c764ccb7079c441aef6045166b4b47cc19c17378bb2d089c38135a9810303"} Nov 02 03:44:50.738440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:50.738310 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="675c764ccb7079c441aef6045166b4b47cc19c17378bb2d089c38135a9810303" Nov 02 03:44:57.270909 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:57.270881 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w"] Nov 02 03:44:57.319787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:57.319760 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad96f4-6466w"] Nov 02 03:44:58.114955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:44:58.114925 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fd83e8-8343-4b93-986f-62ce0aebd375" path="/var/lib/kubelet/pods/76fd83e8-8343-4b93-986f-62ce0aebd375/volumes" Nov 02 03:45:00.156228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156191 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d"] Nov 02 03:45:00.156631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156618 4868 cpu_manager.go:414] "containerMap: removing container" podUID="37ecae1e-6d09-48d9-b01b-2d18898e5364" containerName="revoke-keys" Nov 02 03:45:00.156715 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156632 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ecae1e-6d09-48d9-b01b-2d18898e5364" containerName="revoke-keys" Nov 02 03:45:00.156715 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156652 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" containerName="revoke-keys" Nov 02 03:45:00.156715 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156658 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" containerName="revoke-keys" Nov 02 03:45:00.156840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156749 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ecae1e-6d09-48d9-b01b-2d18898e5364" containerName="revoke-keys" Nov 02 03:45:00.156840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.156762 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" containerName="revoke-keys" Nov 02 03:45:00.161315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.161296 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4"] Nov 02 03:45:00.161468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.161444 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.165021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.165004 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f"] Nov 02 03:45:00.165146 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.165123 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.168788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.168426 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.205793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.205771 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd"] Nov 02 03:45:00.209285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.209270 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8"] Nov 02 03:45:00.209405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.209387 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.212680 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.212662 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.213270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.213252 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:45:00.251808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.251786 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5rh\" (UniqueName: \"kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.251999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.251981 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.265869 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.265845 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d"] Nov 02 03:45:00.306594 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.306557 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4"] Nov 02 03:45:00.312455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.312418 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f"] Nov 02 03:45:00.315875 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.315846 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd"] Nov 02 03:45:00.353513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353480 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.353687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353529 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.353687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353585 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnwd\" (UniqueName: \"kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.353815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353762 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.353863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353840 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdlh\" (UniqueName: \"kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh\") pod \"maas-api-key-cleanup-29893185-pzb5f\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.353918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353897 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5rh\" (UniqueName: \"kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.353986 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.353966 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.354042 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.354009 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893185-pzb5f\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.354308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.354166 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.354308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.354171 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.354308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.354229 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866wp\" (UniqueName: \"kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.359892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.359870 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8"] Nov 02 03:45:00.410503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.410445 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5rh\" (UniqueName: \"kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh\") pod \"maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.455491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455467 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.455609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455500 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdlh\" (UniqueName: \"kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh\") pod \"maas-api-key-cleanup-29893185-pzb5f\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.455609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455540 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.455609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455565 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893185-pzb5f\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.455728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455609 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.455728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455644 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866wp\" (UniqueName: \"kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.455728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455685 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.455891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.455733 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnwd\" (UniqueName: \"kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.456310 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.456285 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.456310 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.456302 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893185-pzb5f\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.456435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.456293 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.456435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.456318 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.476372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.476340 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:00.515604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.515569 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnwd\" (UniqueName: \"kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.515842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.515823 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdlh\" (UniqueName: \"kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh\") pod \"maas-api-key-cleanup-29893185-pzb5f\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.515924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.515876 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.519076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.519056 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866wp\" (UniqueName: \"kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp\") pod \"maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.523488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.523466 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:00.531376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.531328 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:00.783247 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.783166 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:00.788242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.788221 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:00.856868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.856837 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d"] Nov 02 03:45:00.867210 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:00.867184 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd482f5_2b08_447b_8024_0cdf34eaf91b.slice/crio-3cb6873183813a1422f32a2bdecb5e6a7e6986ddee3832b120b5f6c9d0ae3a0c WatchSource:0}: Error finding container 3cb6873183813a1422f32a2bdecb5e6a7e6986ddee3832b120b5f6c9d0ae3a0c: Status 404 returned error can't find the container with id 3cb6873183813a1422f32a2bdecb5e6a7e6986ddee3832b120b5f6c9d0ae3a0c Nov 02 03:45:00.918587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.918563 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8"] Nov 02 03:45:00.919444 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:00.919419 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267c206f_6f14_4cbc_8f3d_61452eeaa275.slice/crio-270fddc75cba8e352cd9a4f85f954e710650f1c0c0fe1cd1dad5c6765cbc3c0c WatchSource:0}: Error finding container 270fddc75cba8e352cd9a4f85f954e710650f1c0c0fe1cd1dad5c6765cbc3c0c: Status 404 returned error can't find the container with id 270fddc75cba8e352cd9a4f85f954e710650f1c0c0fe1cd1dad5c6765cbc3c0c Nov 02 03:45:00.956510 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:00.956486 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd"] Nov 02 03:45:00.957187 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:00.957160 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61afa9b0_cadf_434d_ae8a_6cf96511e729.slice/crio-6c21479271938f8f1b248fa6d0aa9af4ebf21d7633b43d41c49a2d3eb806cdb3 WatchSource:0}: Error finding container 6c21479271938f8f1b248fa6d0aa9af4ebf21d7633b43d41c49a2d3eb806cdb3: Status 404 returned error can't find the container with id 6c21479271938f8f1b248fa6d0aa9af4ebf21d7633b43d41c49a2d3eb806cdb3 Nov 02 03:45:01.159635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.159607 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4"] Nov 02 03:45:01.160174 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:01.160150 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05890ca6_f8a7_4af0_a51f_a5df6bcaf8f2.slice/crio-60a68a66cd38b832abbb73b401aead54d2765dd3658d360cd1542e344f2f41e7 WatchSource:0}: Error finding container 60a68a66cd38b832abbb73b401aead54d2765dd3658d360cd1542e344f2f41e7: Status 404 returned error can't find the container with id 60a68a66cd38b832abbb73b401aead54d2765dd3658d360cd1542e344f2f41e7 Nov 02 03:45:01.165907 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.165880 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f"] Nov 02 03:45:01.166488 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:01.166463 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551bbd7a_755c_44d1_be76_d8872adaed92.slice/crio-64c04d6c876d7f9fd5e53409de151d44ce05934ff262b22d01fbd600d329af47 WatchSource:0}: Error finding container 64c04d6c876d7f9fd5e53409de151d44ce05934ff262b22d01fbd600d329af47: Status 404 returned error can't find the container with id 64c04d6c876d7f9fd5e53409de151d44ce05934ff262b22d01fbd600d329af47 Nov 02 03:45:01.821185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.821094 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" event={"ID":"551bbd7a-755c-44d1-be76-d8872adaed92","Type":"ContainerStarted","Data":"119af07ee7cdf930059436e086f80f545e57e4322e2146023e1fa50edb5acefc"} Nov 02 03:45:01.821185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.821158 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" event={"ID":"551bbd7a-755c-44d1-be76-d8872adaed92","Type":"ContainerStarted","Data":"64c04d6c876d7f9fd5e53409de151d44ce05934ff262b22d01fbd600d329af47"} Nov 02 03:45:01.823055 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.823025 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" event={"ID":"4cd482f5-2b08-447b-8024-0cdf34eaf91b","Type":"ContainerStarted","Data":"cae1b2d77ec8400f86ba1cec84f7ab12ec6856195ea72dfd641a3bf26001dd65"} Nov 02 03:45:01.823205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.823081 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" event={"ID":"4cd482f5-2b08-447b-8024-0cdf34eaf91b","Type":"ContainerStarted","Data":"3cb6873183813a1422f32a2bdecb5e6a7e6986ddee3832b120b5f6c9d0ae3a0c"} Nov 02 03:45:01.824593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.824574 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerStarted","Data":"0487cf186edbe1ec5820971398d21b27c78e74b21b1d7d3c25773bbfb69c01ef"} Nov 02 03:45:01.824670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.824614 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerStarted","Data":"270fddc75cba8e352cd9a4f85f954e710650f1c0c0fe1cd1dad5c6765cbc3c0c"} Nov 02 03:45:01.826187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.826168 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerStarted","Data":"ed9e168015978f626de1e436fb52585bc7581f48965a38fadddc38b2e00ecf01"} Nov 02 03:45:01.826273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.826209 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerStarted","Data":"60a68a66cd38b832abbb73b401aead54d2765dd3658d360cd1542e344f2f41e7"} Nov 02 03:45:01.827905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.827884 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerStarted","Data":"4e8cdcc6e87c57f69527998bf273d2150096e0172b5ecdc8b4b43148c20c8880"} Nov 02 03:45:01.828007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.827923 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerStarted","Data":"6c21479271938f8f1b248fa6d0aa9af4ebf21d7633b43d41c49a2d3eb806cdb3"} Nov 02 03:45:01.934308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:01.934161 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" podStartSLOduration=1.934142813 podStartE2EDuration="1.934142813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:01.93323998 +0000 UTC m=+3443.928940926" watchObservedRunningTime="2026-11-02 03:45:01.934142813 +0000 UTC m=+3443.929843755" Nov 02 03:45:02.049127 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.049077 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" podStartSLOduration=2.049061584 podStartE2EDuration="2.049061584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:02.047567219 +0000 UTC m=+3444.043268162" watchObservedRunningTime="2026-11-02 03:45:02.049061584 +0000 UTC m=+3444.044762548" Nov 02 03:45:02.162656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.162600 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" podStartSLOduration=2.162583993 podStartE2EDuration="2.162583993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:02.158977306 +0000 UTC m=+3444.154678253" watchObservedRunningTime="2026-11-02 03:45:02.162583993 +0000 UTC m=+3444.158284936" Nov 02 03:45:02.275619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.273410 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" podStartSLOduration=2.273388295 podStartE2EDuration="2.273388295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:02.270269765 +0000 UTC m=+3444.265970708" watchObservedRunningTime="2026-11-02 03:45:02.273388295 +0000 UTC m=+3444.269089235" Nov 02 03:45:02.385281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.385227 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" podStartSLOduration=2.385213936 podStartE2EDuration="2.385213936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:02.382870178 +0000 UTC m=+3444.378571122" watchObservedRunningTime="2026-11-02 03:45:02.385213936 +0000 UTC m=+3444.380914879" Nov 02 03:45:02.665899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.665825 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr"] Nov 02 03:45:02.670758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.670734 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.684625 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.684597 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.684776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.684757 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.684859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.684793 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tbd\" (UniqueName: \"kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.722980 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.722951 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr"] Nov 02 03:45:02.735450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.735428 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:45:02.785573 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.785548 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.785690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.785583 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tbd\" (UniqueName: \"kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.785690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.785660 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.786279 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.786258 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.787934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.787911 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.838108 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.838084 4868 generic.go:356] "Generic (PLEG): container finished" podID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerID="ed9e168015978f626de1e436fb52585bc7581f48965a38fadddc38b2e00ecf01" exitCode=60 Nov 02 03:45:02.838214 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.838164 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerDied","Data":"ed9e168015978f626de1e436fb52585bc7581f48965a38fadddc38b2e00ecf01"} Nov 02 03:45:02.838445 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.838428 4868 scope.go:122] "RemoveContainer" containerID="ed9e168015978f626de1e436fb52585bc7581f48965a38fadddc38b2e00ecf01" Nov 02 03:45:02.839917 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.839863 4868 generic.go:356] "Generic (PLEG): container finished" podID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerID="4e8cdcc6e87c57f69527998bf273d2150096e0172b5ecdc8b4b43148c20c8880" exitCode=60 Nov 02 03:45:02.839959 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.839943 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerDied","Data":"4e8cdcc6e87c57f69527998bf273d2150096e0172b5ecdc8b4b43148c20c8880"} Nov 02 03:45:02.840155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.840140 4868 scope.go:122] "RemoveContainer" containerID="4e8cdcc6e87c57f69527998bf273d2150096e0172b5ecdc8b4b43148c20c8880" Nov 02 03:45:02.841696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.841679 4868 generic.go:356] "Generic (PLEG): container finished" podID="551bbd7a-755c-44d1-be76-d8872adaed92" containerID="119af07ee7cdf930059436e086f80f545e57e4322e2146023e1fa50edb5acefc" exitCode=60 Nov 02 03:45:02.842028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.841747 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" event={"ID":"551bbd7a-755c-44d1-be76-d8872adaed92","Type":"ContainerDied","Data":"119af07ee7cdf930059436e086f80f545e57e4322e2146023e1fa50edb5acefc"} Nov 02 03:45:02.842028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.841959 4868 scope.go:122] "RemoveContainer" containerID="119af07ee7cdf930059436e086f80f545e57e4322e2146023e1fa50edb5acefc" Nov 02 03:45:02.843707 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.843687 4868 generic.go:356] "Generic (PLEG): container finished" podID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerID="cae1b2d77ec8400f86ba1cec84f7ab12ec6856195ea72dfd641a3bf26001dd65" exitCode=60 Nov 02 03:45:02.843811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.843763 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" event={"ID":"4cd482f5-2b08-447b-8024-0cdf34eaf91b","Type":"ContainerDied","Data":"cae1b2d77ec8400f86ba1cec84f7ab12ec6856195ea72dfd641a3bf26001dd65"} Nov 02 03:45:02.843942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.843928 4868 scope.go:122] "RemoveContainer" containerID="cae1b2d77ec8400f86ba1cec84f7ab12ec6856195ea72dfd641a3bf26001dd65" Nov 02 03:45:02.845625 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.845605 4868 generic.go:356] "Generic (PLEG): container finished" podID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerID="0487cf186edbe1ec5820971398d21b27c78e74b21b1d7d3c25773bbfb69c01ef" exitCode=60 Nov 02 03:45:02.845743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.845641 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerDied","Data":"0487cf186edbe1ec5820971398d21b27c78e74b21b1d7d3c25773bbfb69c01ef"} Nov 02 03:45:02.845876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.845861 4868 scope.go:122] "RemoveContainer" containerID="0487cf186edbe1ec5820971398d21b27c78e74b21b1d7d3c25773bbfb69c01ef" Nov 02 03:45:02.846302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.846280 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tbd\" (UniqueName: \"kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd\") pod \"maas-api-revoke-sub-68043d42-qq7tr\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:02.985425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:02.985398 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:03.406138 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:03.406099 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e33c5f_0128_4894_8995_1049eccd6e4e.slice/crio-1d632b044f72adb5ffed7f3396d574f29d1a14e2a0a35fbba9765d8e9a5f3904 WatchSource:0}: Error finding container 1d632b044f72adb5ffed7f3396d574f29d1a14e2a0a35fbba9765d8e9a5f3904: Status 404 returned error can't find the container with id 1d632b044f72adb5ffed7f3396d574f29d1a14e2a0a35fbba9765d8e9a5f3904 Nov 02 03:45:03.406964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.406932 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr"] Nov 02 03:45:03.454347 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:03.454309 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:45:03.774280 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:03.774247 4868 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd482f5_2b08_447b_8024_0cdf34eaf91b.slice/crio-f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551bbd7a_755c_44d1_be76_d8872adaed92.slice/crio-conmon-f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216.scope\": RecentStats: unable to find data in memory cache]" Nov 02 03:45:03.854487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.854460 4868 generic.go:356] "Generic (PLEG): container finished" podID="551bbd7a-755c-44d1-be76-d8872adaed92" containerID="f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216" exitCode=60 Nov 02 03:45:03.854591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.854540 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" event={"ID":"551bbd7a-755c-44d1-be76-d8872adaed92","Type":"ContainerDied","Data":"f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216"} Nov 02 03:45:03.854676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.854605 4868 scope.go:122] "RemoveContainer" containerID="119af07ee7cdf930059436e086f80f545e57e4322e2146023e1fa50edb5acefc" Nov 02 03:45:03.854920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.854893 4868 scope.go:122] "RemoveContainer" containerID="f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216" Nov 02 03:45:03.855218 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:03.855194 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893185-pzb5f_odh-ai-gateway-infra(551bbd7a-755c-44d1-be76-d8872adaed92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" Nov 02 03:45:03.856632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.856612 4868 generic.go:356] "Generic (PLEG): container finished" podID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerID="f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6" exitCode=60 Nov 02 03:45:03.856732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.856696 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" event={"ID":"4cd482f5-2b08-447b-8024-0cdf34eaf91b","Type":"ContainerDied","Data":"f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6"} Nov 02 03:45:03.856925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.856911 4868 scope.go:122] "RemoveContainer" containerID="f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6" Nov 02 03:45:03.857180 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:03.857161 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d_odh-ai-gateway-infra(4cd482f5-2b08-447b-8024-0cdf34eaf91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" Nov 02 03:45:03.858212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.858189 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" event={"ID":"d4e33c5f-0128-4894-8995-1049eccd6e4e","Type":"ContainerStarted","Data":"760cbadfedc17ef6fcb03c08a6efd095d4bfe7262363d6c5f83a6c876be2175b"} Nov 02 03:45:03.858298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.858235 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" event={"ID":"d4e33c5f-0128-4894-8995-1049eccd6e4e","Type":"ContainerStarted","Data":"1d632b044f72adb5ffed7f3396d574f29d1a14e2a0a35fbba9765d8e9a5f3904"} Nov 02 03:45:03.860441 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.860390 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerStarted","Data":"c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb"} Nov 02 03:45:03.862546 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.862525 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerStarted","Data":"04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378"} Nov 02 03:45:03.864993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.864861 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerStarted","Data":"b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79"} Nov 02 03:45:03.883065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:03.883045 4868 scope.go:122] "RemoveContainer" containerID="cae1b2d77ec8400f86ba1cec84f7ab12ec6856195ea72dfd641a3bf26001dd65" Nov 02 03:45:04.194200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.194145 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" podStartSLOduration=2.194129364 podStartE2EDuration="2.194129364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:04.192825125 +0000 UTC m=+3446.188526074" watchObservedRunningTime="2026-11-02 03:45:04.194129364 +0000 UTC m=+3446.189830331" Nov 02 03:45:04.278925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.278845 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5"] Nov 02 03:45:04.279143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.279117 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" containerID="cri-o://3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e" gracePeriod=30 Nov 02 03:45:04.872948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.872912 4868 generic.go:356] "Generic (PLEG): container finished" podID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerID="04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378" exitCode=60 Nov 02 03:45:04.873561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.872969 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerDied","Data":"04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378"} Nov 02 03:45:04.873561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.873025 4868 scope.go:122] "RemoveContainer" containerID="ed9e168015978f626de1e436fb52585bc7581f48965a38fadddc38b2e00ecf01" Nov 02 03:45:04.873561 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.873239 4868 scope.go:122] "RemoveContainer" containerID="04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378" Nov 02 03:45:04.873561 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:04.873494 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4_odh-ai-gateway-infra(05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" Nov 02 03:45:04.875427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.875407 4868 generic.go:356] "Generic (PLEG): container finished" podID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerID="b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79" exitCode=60 Nov 02 03:45:04.875517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.875494 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerDied","Data":"b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79"} Nov 02 03:45:04.875687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.875675 4868 scope.go:122] "RemoveContainer" containerID="b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79" Nov 02 03:45:04.875912 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:04.875895 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd_odh-ai-gateway-infra(61afa9b0-cadf-434d-ae8a-6cf96511e729)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" Nov 02 03:45:04.877366 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.877329 4868 scope.go:122] "RemoveContainer" containerID="f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216" Nov 02 03:45:04.877612 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:04.877587 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893185-pzb5f_odh-ai-gateway-infra(551bbd7a-755c-44d1-be76-d8872adaed92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" Nov 02 03:45:04.878828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.878808 4868 scope.go:122] "RemoveContainer" containerID="f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6" Nov 02 03:45:04.879033 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:04.879009 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d_odh-ai-gateway-infra(4cd482f5-2b08-447b-8024-0cdf34eaf91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" Nov 02 03:45:04.880132 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.880111 4868 generic.go:356] "Generic (PLEG): container finished" podID="d4e33c5f-0128-4894-8995-1049eccd6e4e" containerID="760cbadfedc17ef6fcb03c08a6efd095d4bfe7262363d6c5f83a6c876be2175b" exitCode=0 Nov 02 03:45:04.880236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.880182 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" event={"ID":"d4e33c5f-0128-4894-8995-1049eccd6e4e","Type":"ContainerDied","Data":"760cbadfedc17ef6fcb03c08a6efd095d4bfe7262363d6c5f83a6c876be2175b"} Nov 02 03:45:04.882296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.882278 4868 generic.go:356] "Generic (PLEG): container finished" podID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerID="c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb" exitCode=60 Nov 02 03:45:04.882421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.882328 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerDied","Data":"c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb"} Nov 02 03:45:04.882593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.882578 4868 scope.go:122] "RemoveContainer" containerID="c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb" Nov 02 03:45:04.882806 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:04.882786 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8_odh-ai-gateway-infra(267c206f-6f14-4cbc-8f3d-61452eeaa275)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" Nov 02 03:45:04.887260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.887241 4868 scope.go:122] "RemoveContainer" containerID="4e8cdcc6e87c57f69527998bf273d2150096e0172b5ecdc8b4b43148c20c8880" Nov 02 03:45:04.900298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:04.900278 4868 scope.go:122] "RemoveContainer" containerID="0487cf186edbe1ec5820971398d21b27c78e74b21b1d7d3c25773bbfb69c01ef" Nov 02 03:45:05.891174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:05.891139 4868 scope.go:122] "RemoveContainer" containerID="c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb" Nov 02 03:45:05.891621 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:05.891471 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8_odh-ai-gateway-infra(267c206f-6f14-4cbc-8f3d-61452eeaa275)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" Nov 02 03:45:05.892895 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:05.892871 4868 scope.go:122] "RemoveContainer" containerID="04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378" Nov 02 03:45:05.893142 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:05.893110 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4_odh-ai-gateway-infra(05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" Nov 02 03:45:05.894315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:05.894294 4868 scope.go:122] "RemoveContainer" containerID="b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79" Nov 02 03:45:05.894743 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:05.894536 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd_odh-ai-gateway-infra(61afa9b0-cadf-434d-ae8a-6cf96511e729)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" Nov 02 03:45:06.029925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.029900 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:06.227826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.227752 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca\") pod \"d4e33c5f-0128-4894-8995-1049eccd6e4e\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " Nov 02 03:45:06.227826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.227814 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca\") pod \"d4e33c5f-0128-4894-8995-1049eccd6e4e\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " Nov 02 03:45:06.228046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.227851 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd\" (UniqueName: \"kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd\") pod \"d4e33c5f-0128-4894-8995-1049eccd6e4e\" (UID: \"d4e33c5f-0128-4894-8995-1049eccd6e4e\") " Nov 02 03:45:06.228164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.228138 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca" pod "d4e33c5f-0128-4894-8995-1049eccd6e4e" (UID: "d4e33c5f-0128-4894-8995-1049eccd6e4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:06.228309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.228289 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:06.230016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.229993 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca" pod "d4e33c5f-0128-4894-8995-1049eccd6e4e" (UID: "d4e33c5f-0128-4894-8995-1049eccd6e4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:45:06.230144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.230088 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd" pod "d4e33c5f-0128-4894-8995-1049eccd6e4e" (UID: "d4e33c5f-0128-4894-8995-1049eccd6e4e"). InnerVolumeSpecName "kube-api-access-k5tbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:06.328889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.328860 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e33c5f-0128-4894-8995-1049eccd6e4e-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:06.328889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.328886 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tbd\" (UniqueName: \"kubernetes.io/projected/d4e33c5f-0128-4894-8995-1049eccd6e4e-kube-api-access-k5tbd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:06.902451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.902423 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" Nov 02 03:45:06.902840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.902422 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr" event={"ID":"d4e33c5f-0128-4894-8995-1049eccd6e4e","Type":"ContainerDied","Data":"1d632b044f72adb5ffed7f3396d574f29d1a14e2a0a35fbba9765d8e9a5f3904"} Nov 02 03:45:06.902840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:06.902551 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d632b044f72adb5ffed7f3396d574f29d1a14e2a0a35fbba9765d8e9a5f3904" Nov 02 03:45:09.537999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.537975 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:45:09.563264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563213 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563298 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563326 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563382 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563405 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563433 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563483 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563518 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563531 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:45:09.563646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563622 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy\") pod \"9898e751-dc26-4f43-8979-fb1646684f98\" (UID: \"9898e751-dc26-4f43-8979-fb1646684f98\") " Nov 02 03:45:09.563920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563723 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:45:09.563920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563835 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:45:09.563920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563845 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:09.564070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.563913 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:45:09.564139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.564121 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.564139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.564138 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9898e751-dc26-4f43-8979-fb1646684f98-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.564252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.564151 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.564252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.564165 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.564252 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.564177 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.565644 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.565616 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:09.565780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.565766 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:45:09.565998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.565976 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "kube-api-access-rh6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:09.565998 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.565981 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo" pod "9898e751-dc26-4f43-8979-fb1646684f98" (UID: "9898e751-dc26-4f43-8979-fb1646684f98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:45:09.664889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.664861 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9898e751-dc26-4f43-8979-fb1646684f98-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.664889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.664890 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6qd\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-kube-api-access-rh6qd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.665076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.664905 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9898e751-dc26-4f43-8979-fb1646684f98-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.665076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.664914 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9898e751-dc26-4f43-8979-fb1646684f98-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:09.926014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.925982 4868 generic.go:356] "Generic (PLEG): container finished" podID="9898e751-dc26-4f43-8979-fb1646684f98" containerID="3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e" exitCode=0 Nov 02 03:45:09.926210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.926029 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" Nov 02 03:45:09.926210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.926065 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" event={"ID":"9898e751-dc26-4f43-8979-fb1646684f98","Type":"ContainerDied","Data":"3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e"} Nov 02 03:45:09.926210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.926154 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5" event={"ID":"9898e751-dc26-4f43-8979-fb1646684f98","Type":"ContainerDied","Data":"81d726eec21f2224c45ee5aba9c2629d71478c78200253d44d97aec664bae96c"} Nov 02 03:45:09.926210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.926189 4868 scope.go:122] "RemoveContainer" containerID="3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e" Nov 02 03:45:09.937764 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.937747 4868 scope.go:122] "RemoveContainer" containerID="55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1" Nov 02 03:45:09.946824 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.946803 4868 scope.go:122] "RemoveContainer" containerID="3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e" Nov 02 03:45:09.947076 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:09.947054 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e\": container with ID starting with 3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e not found: ID does not exist" containerID="3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e" Nov 02 03:45:09.947164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.947083 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e"} err="failed to get container status \"3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e\": rpc error: code = NotFound desc = could not find container \"3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e\": container with ID starting with 3134c68b34d1a22e6d11bb56ec30f7eab6fc36bdf323f76873a403ac162d550e not found: ID does not exist" Nov 02 03:45:09.947164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.947102 4868 scope.go:122] "RemoveContainer" containerID="55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1" Nov 02 03:45:09.947404 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:09.947344 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1\": container with ID starting with 55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1 not found: ID does not exist" containerID="55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1" Nov 02 03:45:09.947461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:09.947411 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1"} err="failed to get container status \"55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1\": rpc error: code = NotFound desc = could not find container \"55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1\": container with ID starting with 55153b6d73532c02a0c58d98081728952296dbcb7670880f29cc35b035b96dc1 not found: ID does not exist" Nov 02 03:45:10.094049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:10.094017 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5"] Nov 02 03:45:10.097260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:10.097233 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ad96f4-openshift-default-7d4cf57fdf-wknv5"] Nov 02 03:45:10.115075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:10.115045 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9898e751-dc26-4f43-8979-fb1646684f98" path="/var/lib/kubelet/pods/9898e751-dc26-4f43-8979-fb1646684f98/volumes" Nov 02 03:45:13.407843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.407813 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8"] Nov 02 03:45:13.408261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408248 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" Nov 02 03:45:13.408302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408261 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" Nov 02 03:45:13.408302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408273 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d4e33c5f-0128-4894-8995-1049eccd6e4e" containerName="revoke-keys" Nov 02 03:45:13.408302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408278 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e33c5f-0128-4894-8995-1049eccd6e4e" containerName="revoke-keys" Nov 02 03:45:13.408302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408296 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" Nov 02 03:45:13.408302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408301 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" Nov 02 03:45:13.408469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408436 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" Nov 02 03:45:13.408469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.408451 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e33c5f-0128-4894-8995-1049eccd6e4e" containerName="revoke-keys" Nov 02 03:45:13.411823 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.411804 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.466728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.466697 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:45:13.498399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.470874 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8"] Nov 02 03:45:13.506681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.505620 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.506681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.505700 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2dn\" (UniqueName: \"kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.506681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.505873 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.607333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.607296 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2dn\" (UniqueName: \"kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.607554 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.607411 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.607554 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.607442 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.608123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.608101 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.609718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.609695 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.663492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.663406 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2dn\" (UniqueName: \"kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn\") pod \"maas-api-revoke-sub-ab01c625-7wrl8\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:13.725206 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:13.725168 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:14.123065 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.123040 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8"] Nov 02 03:45:14.155101 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.155071 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx"] Nov 02 03:45:14.156098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.156074 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9898e751-dc26-4f43-8979-fb1646684f98" containerName="istio-proxy" Nov 02 03:45:14.160806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.160784 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.213732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.213700 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.213838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.213755 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz22\" (UniqueName: \"kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.213896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.213843 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.230488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.230464 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx"] Nov 02 03:45:14.315428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.315400 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.315562 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.315441 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz22\" (UniqueName: \"kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.315562 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.315480 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.316141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.316119 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.332674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.332649 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.372725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.372663 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz22\" (UniqueName: \"kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22\") pod \"maas-api-revoke-sub-ade77231-kjbbx\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.481377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.476267 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:14.891443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.891413 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx"] Nov 02 03:45:14.892336 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:14.892308 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17049507_8137_4f52_a24a_2ce6fe0715a2.slice/crio-fba62121c7d1ad3dbffbbfd70056a511ab699a5f1a83eb510e0e7f907f2c3448 WatchSource:0}: Error finding container fba62121c7d1ad3dbffbbfd70056a511ab699a5f1a83eb510e0e7f907f2c3448: Status 404 returned error can't find the container with id fba62121c7d1ad3dbffbbfd70056a511ab699a5f1a83eb510e0e7f907f2c3448 Nov 02 03:45:14.960609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.960581 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" event={"ID":"17049507-8137-4f52-a24a-2ce6fe0715a2","Type":"ContainerStarted","Data":"fba62121c7d1ad3dbffbbfd70056a511ab699a5f1a83eb510e0e7f907f2c3448"} Nov 02 03:45:14.962379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.962328 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" event={"ID":"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e","Type":"ContainerStarted","Data":"4c4f77ac87e576c55f6c793b249c97661ee7a664ab2a3af00bd3abbc4afc52d3"} Nov 02 03:45:14.962473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:14.962389 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" event={"ID":"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e","Type":"ContainerStarted","Data":"13d45c3fbb91b977662b6a52c080fd5b4408cab6d349ca246c9798e3ee4ee060"} Nov 02 03:45:15.076551 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:15.076501 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" podStartSLOduration=2.076484884 podStartE2EDuration="2.076484884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:15.075652219 +0000 UTC m=+3457.071353162" watchObservedRunningTime="2026-11-02 03:45:15.076484884 +0000 UTC m=+3457.072185829" Nov 02 03:45:15.971154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:15.971106 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" event={"ID":"17049507-8137-4f52-a24a-2ce6fe0715a2","Type":"ContainerStarted","Data":"69631582e6bd28ff946def39f1efecda2f7496cc0f04ebed014584a9101f9245"} Nov 02 03:45:15.972838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:15.972805 4868 generic.go:356] "Generic (PLEG): container finished" podID="1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" containerID="4c4f77ac87e576c55f6c793b249c97661ee7a664ab2a3af00bd3abbc4afc52d3" exitCode=0 Nov 02 03:45:15.972988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:15.972888 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" event={"ID":"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e","Type":"ContainerDied","Data":"4c4f77ac87e576c55f6c793b249c97661ee7a664ab2a3af00bd3abbc4afc52d3"} Nov 02 03:45:16.086418 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.086365 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" podStartSLOduration=2.086341611 podStartE2EDuration="2.086341611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:16.082932383 +0000 UTC m=+3458.078633317" watchObservedRunningTime="2026-11-02 03:45:16.086341611 +0000 UTC m=+3458.082042553" Nov 02 03:45:16.107862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.107833 4868 scope.go:122] "RemoveContainer" containerID="f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6" Nov 02 03:45:16.982376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.982331 4868 generic.go:356] "Generic (PLEG): container finished" podID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerID="bae9ff021c0bd1d22e8f99c89e91a3959dcbb1e4949d180bb01f64770d51443e" exitCode=60 Nov 02 03:45:16.982808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.982394 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" event={"ID":"4cd482f5-2b08-447b-8024-0cdf34eaf91b","Type":"ContainerDied","Data":"bae9ff021c0bd1d22e8f99c89e91a3959dcbb1e4949d180bb01f64770d51443e"} Nov 02 03:45:16.982808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.982458 4868 scope.go:122] "RemoveContainer" containerID="f08e993536e43f85fbf4d905d96fce4770007587686b8ccb640a510884b580a6" Nov 02 03:45:16.982918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.982868 4868 scope.go:122] "RemoveContainer" containerID="bae9ff021c0bd1d22e8f99c89e91a3959dcbb1e4949d180bb01f64770d51443e" Nov 02 03:45:16.983170 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:16.983149 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d_odh-ai-gateway-infra(4cd482f5-2b08-447b-8024-0cdf34eaf91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" Nov 02 03:45:16.985106 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.985086 4868 generic.go:356] "Generic (PLEG): container finished" podID="17049507-8137-4f52-a24a-2ce6fe0715a2" containerID="69631582e6bd28ff946def39f1efecda2f7496cc0f04ebed014584a9101f9245" exitCode=0 Nov 02 03:45:16.985212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:16.985165 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" event={"ID":"17049507-8137-4f52-a24a-2ce6fe0715a2","Type":"ContainerDied","Data":"69631582e6bd28ff946def39f1efecda2f7496cc0f04ebed014584a9101f9245"} Nov 02 03:45:17.108333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.108308 4868 scope.go:122] "RemoveContainer" containerID="f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216" Nov 02 03:45:17.108479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.108462 4868 scope.go:122] "RemoveContainer" containerID="b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79" Nov 02 03:45:17.152625 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.152604 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:17.249037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.248980 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn\" (UniqueName: \"kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn\") pod \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " Nov 02 03:45:17.249037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.249034 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca\") pod \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " Nov 02 03:45:17.249179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.249063 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca\") pod \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\" (UID: \"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e\") " Nov 02 03:45:17.249509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.249485 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca" pod "1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" (UID: "1eb5e2a6-a7fc-468f-a9d9-925b7976d31e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:17.249610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.249564 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:17.250957 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.250934 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn" pod "1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" (UID: "1eb5e2a6-a7fc-468f-a9d9-925b7976d31e"). InnerVolumeSpecName "kube-api-access-6n2dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:17.251056 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.251004 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca" pod "1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" (UID: "1eb5e2a6-a7fc-468f-a9d9-925b7976d31e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:45:17.354559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.350569 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:17.354559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.350596 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2dn\" (UniqueName: \"kubernetes.io/projected/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e-kube-api-access-6n2dn\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:17.995395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.995363 4868 generic.go:356] "Generic (PLEG): container finished" podID="551bbd7a-755c-44d1-be76-d8872adaed92" containerID="f303f7c33edf73ae0bf77a78db14132df5e49a82ee3086fcff6a596d3f38cc84" exitCode=60 Nov 02 03:45:17.995395 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.995380 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" event={"ID":"551bbd7a-755c-44d1-be76-d8872adaed92","Type":"ContainerDied","Data":"f303f7c33edf73ae0bf77a78db14132df5e49a82ee3086fcff6a596d3f38cc84"} Nov 02 03:45:17.995991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.995448 4868 scope.go:122] "RemoveContainer" containerID="f6b088d9c08d2e0387a1ad452a9eddc41b3b494bb78ea37dfb923853c11fe216" Nov 02 03:45:17.995991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.995904 4868 scope.go:122] "RemoveContainer" containerID="f303f7c33edf73ae0bf77a78db14132df5e49a82ee3086fcff6a596d3f38cc84" Nov 02 03:45:17.996210 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:17.996192 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893185-pzb5f_odh-ai-gateway-infra(551bbd7a-755c-44d1-be76-d8872adaed92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" Nov 02 03:45:17.998654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.998631 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" Nov 02 03:45:17.998744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.998676 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8" event={"ID":"1eb5e2a6-a7fc-468f-a9d9-925b7976d31e","Type":"ContainerDied","Data":"13d45c3fbb91b977662b6a52c080fd5b4408cab6d349ca246c9798e3ee4ee060"} Nov 02 03:45:17.998744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:17.998715 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13d45c3fbb91b977662b6a52c080fd5b4408cab6d349ca246c9798e3ee4ee060" Nov 02 03:45:18.000817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.000792 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerStarted","Data":"d4c61c0e2cfbc51fddaa17535f1e0d4a3a2a8045341a509dc7c60af3daf13e3e"} Nov 02 03:45:18.108814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.108778 4868 scope.go:122] "RemoveContainer" containerID="c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb" Nov 02 03:45:18.116968 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.109953 4868 scope.go:122] "RemoveContainer" containerID="04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378" Nov 02 03:45:18.120075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.120045 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d"] Nov 02 03:45:18.242583 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.242541 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:18.270567 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.270547 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:18.360928 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.360898 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh\" (UniqueName: \"kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh\") pod \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " Nov 02 03:45:18.361114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.360968 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca\") pod \"17049507-8137-4f52-a24a-2ce6fe0715a2\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " Nov 02 03:45:18.361114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.361035 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22\" (UniqueName: \"kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22\") pod \"17049507-8137-4f52-a24a-2ce6fe0715a2\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " Nov 02 03:45:18.361246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.361139 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca\") pod \"17049507-8137-4f52-a24a-2ce6fe0715a2\" (UID: \"17049507-8137-4f52-a24a-2ce6fe0715a2\") " Nov 02 03:45:18.361246 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.361190 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca\") pod \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\" (UID: \"4cd482f5-2b08-447b-8024-0cdf34eaf91b\") " Nov 02 03:45:18.361900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.361528 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca" pod "17049507-8137-4f52-a24a-2ce6fe0715a2" (UID: "17049507-8137-4f52-a24a-2ce6fe0715a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:18.361900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.361690 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:18.361900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.361857 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca" pod "4cd482f5-2b08-447b-8024-0cdf34eaf91b" (UID: "4cd482f5-2b08-447b-8024-0cdf34eaf91b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:18.363177 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.363151 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca" pod "17049507-8137-4f52-a24a-2ce6fe0715a2" (UID: "17049507-8137-4f52-a24a-2ce6fe0715a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:45:18.363276 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.363158 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22" pod "17049507-8137-4f52-a24a-2ce6fe0715a2" (UID: "17049507-8137-4f52-a24a-2ce6fe0715a2"). InnerVolumeSpecName "kube-api-access-cqz22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:18.363503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.363485 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh" pod "4cd482f5-2b08-447b-8024-0cdf34eaf91b" (UID: "4cd482f5-2b08-447b-8024-0cdf34eaf91b"). InnerVolumeSpecName "kube-api-access-kj5rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:18.463215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.463190 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd482f5-2b08-447b-8024-0cdf34eaf91b-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:18.463215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.463214 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5rh\" (UniqueName: \"kubernetes.io/projected/4cd482f5-2b08-447b-8024-0cdf34eaf91b-kube-api-access-kj5rh\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:18.463390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.463224 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17049507-8137-4f52-a24a-2ce6fe0715a2-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:18.463390 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:18.463234 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqz22\" (UniqueName: \"kubernetes.io/projected/17049507-8137-4f52-a24a-2ce6fe0715a2-kube-api-access-cqz22\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:19.010270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.010243 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" Nov 02 03:45:19.010756 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.010266 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d" event={"ID":"4cd482f5-2b08-447b-8024-0cdf34eaf91b","Type":"ContainerDied","Data":"3cb6873183813a1422f32a2bdecb5e6a7e6986ddee3832b120b5f6c9d0ae3a0c"} Nov 02 03:45:19.010756 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.010328 4868 scope.go:122] "RemoveContainer" containerID="bae9ff021c0bd1d22e8f99c89e91a3959dcbb1e4949d180bb01f64770d51443e" Nov 02 03:45:19.011651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.011627 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" Nov 02 03:45:19.011651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.011635 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx" event={"ID":"17049507-8137-4f52-a24a-2ce6fe0715a2","Type":"ContainerDied","Data":"fba62121c7d1ad3dbffbbfd70056a511ab699a5f1a83eb510e0e7f907f2c3448"} Nov 02 03:45:19.011798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.011675 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba62121c7d1ad3dbffbbfd70056a511ab699a5f1a83eb510e0e7f907f2c3448" Nov 02 03:45:19.013925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.013902 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerStarted","Data":"a987f7fa5a1efb2a1a15677f2f9b37ce5996787c48be2aba748e705b3d0ff6b7"} Nov 02 03:45:19.016011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.015987 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerStarted","Data":"fefe589da18ff9dea4abf27489e2e919a10378a00ffdfef6eb0833f9df66a651"} Nov 02 03:45:19.018073 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.018051 4868 generic.go:356] "Generic (PLEG): container finished" podID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerID="d4c61c0e2cfbc51fddaa17535f1e0d4a3a2a8045341a509dc7c60af3daf13e3e" exitCode=60 Nov 02 03:45:19.018152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.018120 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerDied","Data":"d4c61c0e2cfbc51fddaa17535f1e0d4a3a2a8045341a509dc7c60af3daf13e3e"} Nov 02 03:45:19.018461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.018445 4868 scope.go:122] "RemoveContainer" containerID="d4c61c0e2cfbc51fddaa17535f1e0d4a3a2a8045341a509dc7c60af3daf13e3e" Nov 02 03:45:19.018709 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:19.018688 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd_odh-ai-gateway-infra(61afa9b0-cadf-434d-ae8a-6cf96511e729)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" Nov 02 03:45:19.022098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.022075 4868 scope.go:122] "RemoveContainer" containerID="b071a6d42783e3d1c2d19693c9a4eb5b30f975438e5a6fa156fea9af8428dd79" Nov 02 03:45:19.123627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.123600 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f"] Nov 02 03:45:19.228037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.228009 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d"] Nov 02 03:45:19.236361 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.236333 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5ac93f-29893185-2jp8d"] Nov 02 03:45:19.284646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.284581 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd"] Nov 02 03:45:19.352267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.352246 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:19.474862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.474832 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca\") pod \"551bbd7a-755c-44d1-be76-d8872adaed92\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " Nov 02 03:45:19.475012 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.474931 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh\" (UniqueName: \"kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh\") pod \"551bbd7a-755c-44d1-be76-d8872adaed92\" (UID: \"551bbd7a-755c-44d1-be76-d8872adaed92\") " Nov 02 03:45:19.475192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.475160 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca" pod "551bbd7a-755c-44d1-be76-d8872adaed92" (UID: "551bbd7a-755c-44d1-be76-d8872adaed92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:19.475433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.475414 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/551bbd7a-755c-44d1-be76-d8872adaed92-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:19.476893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.476875 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh" pod "551bbd7a-755c-44d1-be76-d8872adaed92" (UID: "551bbd7a-755c-44d1-be76-d8872adaed92"). InnerVolumeSpecName "kube-api-access-4pdlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:19.576985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:19.576915 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdlh\" (UniqueName: \"kubernetes.io/projected/551bbd7a-755c-44d1-be76-d8872adaed92-kube-api-access-4pdlh\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:20.026826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.026791 4868 generic.go:356] "Generic (PLEG): container finished" podID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerID="a987f7fa5a1efb2a1a15677f2f9b37ce5996787c48be2aba748e705b3d0ff6b7" exitCode=60 Nov 02 03:45:20.027278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.026855 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerDied","Data":"a987f7fa5a1efb2a1a15677f2f9b37ce5996787c48be2aba748e705b3d0ff6b7"} Nov 02 03:45:20.027278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.026917 4868 scope.go:122] "RemoveContainer" containerID="c5e6d464fb1f63107acf55c3a6c90d330a6cd78a748ac7b345b671b82f5a21fb" Nov 02 03:45:20.027278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.027185 4868 scope.go:122] "RemoveContainer" containerID="a987f7fa5a1efb2a1a15677f2f9b37ce5996787c48be2aba748e705b3d0ff6b7" Nov 02 03:45:20.027498 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:20.027473 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8_odh-ai-gateway-infra(267c206f-6f14-4cbc-8f3d-61452eeaa275)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" Nov 02 03:45:20.029236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.029210 4868 generic.go:356] "Generic (PLEG): container finished" podID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerID="fefe589da18ff9dea4abf27489e2e919a10378a00ffdfef6eb0833f9df66a651" exitCode=60 Nov 02 03:45:20.029327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.029241 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerDied","Data":"fefe589da18ff9dea4abf27489e2e919a10378a00ffdfef6eb0833f9df66a651"} Nov 02 03:45:20.029576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.029557 4868 scope.go:122] "RemoveContainer" containerID="fefe589da18ff9dea4abf27489e2e919a10378a00ffdfef6eb0833f9df66a651" Nov 02 03:45:20.029816 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:20.029792 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4_odh-ai-gateway-infra(05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" Nov 02 03:45:20.032517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.032493 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" Nov 02 03:45:20.032606 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.032523 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f" event={"ID":"551bbd7a-755c-44d1-be76-d8872adaed92","Type":"ContainerDied","Data":"64c04d6c876d7f9fd5e53409de151d44ce05934ff262b22d01fbd600d329af47"} Nov 02 03:45:20.042181 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.042150 4868 scope.go:122] "RemoveContainer" containerID="04e7d4e333d0c122ade735b5c33c93f1c856b4db46ae0a383a1f634db0b50378" Nov 02 03:45:20.055654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.055637 4868 scope.go:122] "RemoveContainer" containerID="f303f7c33edf73ae0bf77a78db14132df5e49a82ee3086fcff6a596d3f38cc84" Nov 02 03:45:20.117196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.117168 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" path="/var/lib/kubelet/pods/4cd482f5-2b08-447b-8024-0cdf34eaf91b/volumes" Nov 02 03:45:20.186534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.186510 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:20.250767 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.250741 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f"] Nov 02 03:45:20.253597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.253578 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893185-pzb5f"] Nov 02 03:45:20.286690 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.286628 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt\") pod \"61afa9b0-cadf-434d-ae8a-6cf96511e729\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " Nov 02 03:45:20.286825 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.286805 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca\") pod \"61afa9b0-cadf-434d-ae8a-6cf96511e729\" (UID: \"61afa9b0-cadf-434d-ae8a-6cf96511e729\") " Nov 02 03:45:20.287154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.287128 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca" pod "61afa9b0-cadf-434d-ae8a-6cf96511e729" (UID: "61afa9b0-cadf-434d-ae8a-6cf96511e729"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:20.288707 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.288688 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt" pod "61afa9b0-cadf-434d-ae8a-6cf96511e729" (UID: "61afa9b0-cadf-434d-ae8a-6cf96511e729"). InnerVolumeSpecName "kube-api-access-5cvtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:20.360464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.360436 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8"] Nov 02 03:45:20.387786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.387760 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvtt\" (UniqueName: \"kubernetes.io/projected/61afa9b0-cadf-434d-ae8a-6cf96511e729-kube-api-access-5cvtt\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:20.387786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.387784 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61afa9b0-cadf-434d-ae8a-6cf96511e729-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:20.468984 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.468954 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4"] Nov 02 03:45:20.855743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.855713 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h"] Nov 02 03:45:20.856154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856142 4868 cpu_manager.go:414] "containerMap: removing container" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:20.856154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856156 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856168 4868 cpu_manager.go:414] "containerMap: removing container" podUID="1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" containerName="revoke-keys" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856173 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" containerName="revoke-keys" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856183 4868 cpu_manager.go:414] "containerMap: removing container" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856188 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856199 4868 cpu_manager.go:414] "containerMap: removing container" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856203 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856209 4868 cpu_manager.go:414] "containerMap: removing container" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856213 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856221 4868 cpu_manager.go:414] "containerMap: removing container" podUID="17049507-8137-4f52-a24a-2ce6fe0715a2" containerName="revoke-keys" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856225 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17049507-8137-4f52-a24a-2ce6fe0715a2" containerName="revoke-keys" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856232 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:20.856239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856238 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856267 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856271 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856346 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856370 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856379 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="17049507-8137-4f52-a24a-2ce6fe0715a2" containerName="revoke-keys" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856391 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856404 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856410 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856416 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856422 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" containerName="revoke-keys" Nov 02 03:45:20.856621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.856427 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:20.859950 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.859931 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:20.912090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.912068 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:45:20.915893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.915870 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h"] Nov 02 03:45:20.994539 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.994512 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:20.994658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.994582 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngtb\" (UniqueName: \"kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:20.994698 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:20.994658 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.046594 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.046577 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" Nov 02 03:45:21.046975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.046619 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd" event={"ID":"61afa9b0-cadf-434d-ae8a-6cf96511e729","Type":"ContainerDied","Data":"6c21479271938f8f1b248fa6d0aa9af4ebf21d7633b43d41c49a2d3eb806cdb3"} Nov 02 03:45:21.046975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.046670 4868 scope.go:122] "RemoveContainer" containerID="d4c61c0e2cfbc51fddaa17535f1e0d4a3a2a8045341a509dc7c60af3daf13e3e" Nov 02 03:45:21.095487 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.095408 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.095670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.095579 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngtb\" (UniqueName: \"kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.096090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.096062 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.096819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.096793 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.098758 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.098731 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.152330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.152301 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngtb\" (UniqueName: \"kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb\") pod \"maas-api-revoke-sub-a8bcaa7a-v6d5h\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.173161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.173127 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:21.241922 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.241852 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:21.248117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.248091 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:21.298117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.298085 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp\" (UniqueName: \"kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp\") pod \"267c206f-6f14-4cbc-8f3d-61452eeaa275\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " Nov 02 03:45:21.298242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.298139 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd\" (UniqueName: \"kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd\") pod \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " Nov 02 03:45:21.298242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.298168 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca\") pod \"267c206f-6f14-4cbc-8f3d-61452eeaa275\" (UID: \"267c206f-6f14-4cbc-8f3d-61452eeaa275\") " Nov 02 03:45:21.298387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.298293 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca\") pod \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\" (UID: \"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2\") " Nov 02 03:45:21.298657 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.298588 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca" pod "267c206f-6f14-4cbc-8f3d-61452eeaa275" (UID: "267c206f-6f14-4cbc-8f3d-61452eeaa275"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:21.298813 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.298788 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca" pod "05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" (UID: "05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:21.300436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.300407 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd" pod "05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" (UID: "05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2"). InnerVolumeSpecName "kube-api-access-5xnwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:21.300563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.300538 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp" pod "267c206f-6f14-4cbc-8f3d-61452eeaa275" (UID: "267c206f-6f14-4cbc-8f3d-61452eeaa275"). InnerVolumeSpecName "kube-api-access-866wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:21.379577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.379520 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd"] Nov 02 03:45:21.384455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.384434 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893185-5wshd"] Nov 02 03:45:21.399819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.399798 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866wp\" (UniqueName: \"kubernetes.io/projected/267c206f-6f14-4cbc-8f3d-61452eeaa275-kube-api-access-866wp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:21.399819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.399818 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnwd\" (UniqueName: \"kubernetes.io/projected/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-kube-api-access-5xnwd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:21.399933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.399828 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/267c206f-6f14-4cbc-8f3d-61452eeaa275-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:21.399933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.399840 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:21.560946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:21.560921 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h"] Nov 02 03:45:21.561811 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:21.561782 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd045d06c_4439_4f74_9f4c_e0904736433f.slice/crio-62faf1c30456057e94fffbec46bce41e425d915658f94fe48cd01ae9757f1d30 WatchSource:0}: Error finding container 62faf1c30456057e94fffbec46bce41e425d915658f94fe48cd01ae9757f1d30: Status 404 returned error can't find the container with id 62faf1c30456057e94fffbec46bce41e425d915658f94fe48cd01ae9757f1d30 Nov 02 03:45:22.061928 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.061851 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" Nov 02 03:45:22.062347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.061843 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4" event={"ID":"05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2","Type":"ContainerDied","Data":"60a68a66cd38b832abbb73b401aead54d2765dd3658d360cd1542e344f2f41e7"} Nov 02 03:45:22.062347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.062020 4868 scope.go:122] "RemoveContainer" containerID="fefe589da18ff9dea4abf27489e2e919a10378a00ffdfef6eb0833f9df66a651" Nov 02 03:45:22.064961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.064931 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" event={"ID":"d045d06c-4439-4f74-9f4c-e0904736433f","Type":"ContainerStarted","Data":"da6eee9bf2fcacd235e6dbe5ad8ea612ca3d82ddc175e25c3cb7216b83b4fd7c"} Nov 02 03:45:22.065093 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.064988 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" event={"ID":"d045d06c-4439-4f74-9f4c-e0904736433f","Type":"ContainerStarted","Data":"62faf1c30456057e94fffbec46bce41e425d915658f94fe48cd01ae9757f1d30"} Nov 02 03:45:22.066342 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.066321 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" event={"ID":"267c206f-6f14-4cbc-8f3d-61452eeaa275","Type":"ContainerDied","Data":"270fddc75cba8e352cd9a4f85f954e710650f1c0c0fe1cd1dad5c6765cbc3c0c"} Nov 02 03:45:22.066444 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.066365 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8" Nov 02 03:45:22.073026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.073010 4868 scope.go:122] "RemoveContainer" containerID="a987f7fa5a1efb2a1a15677f2f9b37ce5996787c48be2aba748e705b3d0ff6b7" Nov 02 03:45:22.113703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.113683 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" path="/var/lib/kubelet/pods/551bbd7a-755c-44d1-be76-d8872adaed92/volumes" Nov 02 03:45:22.114117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.114102 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" path="/var/lib/kubelet/pods/61afa9b0-cadf-434d-ae8a-6cf96511e729/volumes" Nov 02 03:45:22.224727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.224701 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4"] Nov 02 03:45:22.228571 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.228550 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893185-pqsz4"] Nov 02 03:45:22.343301 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.343220 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" podStartSLOduration=2.343208058 podStartE2EDuration="2.343208058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:22.339244218 +0000 UTC m=+3464.334945160" watchObservedRunningTime="2026-11-02 03:45:22.343208058 +0000 UTC m=+3464.338909001" Nov 02 03:45:22.450410 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.450380 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8"] Nov 02 03:45:22.453302 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:22.453279 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ca6546-29893185-g8dh8"] Nov 02 03:45:23.074582 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:23.074546 4868 generic.go:356] "Generic (PLEG): container finished" podID="d045d06c-4439-4f74-9f4c-e0904736433f" containerID="da6eee9bf2fcacd235e6dbe5ad8ea612ca3d82ddc175e25c3cb7216b83b4fd7c" exitCode=0 Nov 02 03:45:23.075017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:23.074627 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" event={"ID":"d045d06c-4439-4f74-9f4c-e0904736433f","Type":"ContainerDied","Data":"da6eee9bf2fcacd235e6dbe5ad8ea612ca3d82ddc175e25c3cb7216b83b4fd7c"} Nov 02 03:45:24.116420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.116384 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" path="/var/lib/kubelet/pods/05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2/volumes" Nov 02 03:45:24.117020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.116999 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" path="/var/lib/kubelet/pods/267c206f-6f14-4cbc-8f3d-61452eeaa275/volumes" Nov 02 03:45:24.239365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.239326 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:24.328008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.327979 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca\") pod \"d045d06c-4439-4f74-9f4c-e0904736433f\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " Nov 02 03:45:24.328147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.328066 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca\") pod \"d045d06c-4439-4f74-9f4c-e0904736433f\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " Nov 02 03:45:24.328147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.328132 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb\" (UniqueName: \"kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb\") pod \"d045d06c-4439-4f74-9f4c-e0904736433f\" (UID: \"d045d06c-4439-4f74-9f4c-e0904736433f\") " Nov 02 03:45:24.328443 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.328415 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca" pod "d045d06c-4439-4f74-9f4c-e0904736433f" (UID: "d045d06c-4439-4f74-9f4c-e0904736433f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:45:24.330143 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.330122 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb" pod "d045d06c-4439-4f74-9f4c-e0904736433f" (UID: "d045d06c-4439-4f74-9f4c-e0904736433f"). InnerVolumeSpecName "kube-api-access-6ngtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:45:24.330222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.330139 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca" pod "d045d06c-4439-4f74-9f4c-e0904736433f" (UID: "d045d06c-4439-4f74-9f4c-e0904736433f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:45:24.429786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.429755 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngtb\" (UniqueName: \"kubernetes.io/projected/d045d06c-4439-4f74-9f4c-e0904736433f-kube-api-access-6ngtb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:24.429786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.429784 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:24.429970 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:24.429794 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045d06c-4439-4f74-9f4c-e0904736433f-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:45:25.059318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059283 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l"] Nov 02 03:45:25.059862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059831 4868 cpu_manager.go:414] "containerMap: removing container" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:25.059862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059853 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059873 4868 cpu_manager.go:414] "containerMap: removing container" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059878 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059887 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d045d06c-4439-4f74-9f4c-e0904736433f" containerName="revoke-keys" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059892 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d045d06c-4439-4f74-9f4c-e0904736433f" containerName="revoke-keys" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059911 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059916 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059923 4868 cpu_manager.go:414] "containerMap: removing container" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059929 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059945 4868 cpu_manager.go:414] "containerMap: removing container" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059952 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61afa9b0-cadf-434d-ae8a-6cf96511e729" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059962 4868 cpu_manager.go:414] "containerMap: removing container" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059969 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059981 4868 cpu_manager.go:414] "containerMap: removing container" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059988 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.059999 4868 cpu_manager.go:414] "containerMap: removing container" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060003 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060085 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060093 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d045d06c-4439-4f74-9f4c-e0904736433f" containerName="revoke-keys" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060103 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060108 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060115 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="551bbd7a-755c-44d1-be76-d8872adaed92" containerName="cleanup" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060122 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:25.060508 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.060127 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd482f5-2b08-447b-8024-0cdf34eaf91b" containerName="cleanup" Nov 02 03:45:25.063485 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.063469 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.092977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.092952 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" event={"ID":"d045d06c-4439-4f74-9f4c-e0904736433f","Type":"ContainerDied","Data":"62faf1c30456057e94fffbec46bce41e425d915658f94fe48cd01ae9757f1d30"} Nov 02 03:45:25.092977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.092973 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h" Nov 02 03:45:25.093121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.093000 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62faf1c30456057e94fffbec46bce41e425d915658f94fe48cd01ae9757f1d30" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.164622 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ee0fed-openshift-default-dockercfg-p8dck\"" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165130 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165159 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165187 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165212 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165244 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165264 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k889w\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165297 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165341 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.173374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.165500 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.200417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.197641 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l"] Nov 02 03:45:25.266576 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.266545 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.266735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.266587 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.266836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.266815 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.266900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.266882 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.266949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.266909 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.266993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.266950 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267047 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267089 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267165 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267149 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267224 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267180 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k889w\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267264 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267433 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267583 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.267794 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.267773 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.269344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.269320 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.269446 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.269407 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.329625 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.329547 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k889w\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.338437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.338399 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token\") pod \"e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.381022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.380991 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:25.815248 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:25.815198 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968ffdd6_532a_4921_a7cc_b9a3a940580d.slice/crio-41fca3369a304868fc8b2451ed829014b1f4de2108026b12fb5edabd95bde43c WatchSource:0}: Error finding container 41fca3369a304868fc8b2451ed829014b1f4de2108026b12fb5edabd95bde43c: Status 404 returned error can't find the container with id 41fca3369a304868fc8b2451ed829014b1f4de2108026b12fb5edabd95bde43c Nov 02 03:45:25.815747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.815722 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l"] Nov 02 03:45:25.818599 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.818564 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:25.818665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.818646 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:25.818726 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:25.818710 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:26.102781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:26.102748 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" event={"ID":"968ffdd6-532a-4921-a7cc-b9a3a940580d","Type":"ContainerStarted","Data":"2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398"} Nov 02 03:45:26.102956 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:26.102814 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" event={"ID":"968ffdd6-532a-4921-a7cc-b9a3a940580d","Type":"ContainerStarted","Data":"41fca3369a304868fc8b2451ed829014b1f4de2108026b12fb5edabd95bde43c"} Nov 02 03:45:26.219311 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:26.219255 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podStartSLOduration=1.219242463 podStartE2EDuration="1.219242463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:26.217410939 +0000 UTC m=+3468.213111885" watchObservedRunningTime="2026-11-02 03:45:26.219242463 +0000 UTC m=+3468.214943406" Nov 02 03:45:26.381335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:26.381255 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:26.387254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:26.387216 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:27.109637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.109600 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:27.111288 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.111259 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:45:27.422270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.422192 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr"] Nov 02 03:45:27.422746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.422730 4868 cpu_manager.go:414] "containerMap: removing container" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:27.422805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.422747 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:27.422867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.422858 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="267c206f-6f14-4cbc-8f3d-61452eeaa275" containerName="cleanup" Nov 02 03:45:27.422903 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.422872 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="05890ca6-f8a7-4af0-a51f-a5df6bcaf8f2" containerName="cleanup" Nov 02 03:45:27.426435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.426418 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.506375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.505651 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-7c1bee-openshift-default-dockercfg-xq26z\"" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.522924 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523013 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523046 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523086 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523113 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523175 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523211 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523318 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwm6\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.529373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.523344 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.545386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.542405 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr"] Nov 02 03:45:27.624650 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.624600 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwm6\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.624834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.624658 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.624834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.624720 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.624952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.624787 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.624952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.624943 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.625112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.625091 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.625939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.625222 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.625939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.625728 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.625939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.625780 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.625939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.625819 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.626285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.626082 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.626285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.626092 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.626413 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.626302 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.627025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.626993 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.630673 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.630651 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.631532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.631509 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.681804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.681737 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.681926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.681874 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwm6\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6\") pod \"e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:27.740531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:27.740495 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:28.206318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:28.206289 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr"] Nov 02 03:45:28.207173 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:28.206900 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb801bbf_9cd2_489e_985f_cd17ef2638de.slice/crio-7ef0e464a5ccf2a4cc152b9689751918860f90390732e0d1aa9d201f93b97595 WatchSource:0}: Error finding container 7ef0e464a5ccf2a4cc152b9689751918860f90390732e0d1aa9d201f93b97595: Status 404 returned error can't find the container with id 7ef0e464a5ccf2a4cc152b9689751918860f90390732e0d1aa9d201f93b97595 Nov 02 03:45:28.209566 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:28.209537 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:28.209654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:28.209595 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:28.209654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:28.209627 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:29.134805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:29.134766 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" event={"ID":"bb801bbf-9cd2-489e-985f-cd17ef2638de","Type":"ContainerStarted","Data":"55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386"} Nov 02 03:45:29.135386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:29.134866 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" event={"ID":"bb801bbf-9cd2-489e-985f-cd17ef2638de","Type":"ContainerStarted","Data":"7ef0e464a5ccf2a4cc152b9689751918860f90390732e0d1aa9d201f93b97595"} Nov 02 03:45:29.355844 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:29.355791 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podStartSLOduration=2.355778563 podStartE2EDuration="2.355778563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:29.303675086 +0000 UTC m=+3471.299376032" watchObservedRunningTime="2026-11-02 03:45:29.355778563 +0000 UTC m=+3471.351479506" Nov 02 03:45:29.741346 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:29.741309 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:29.746834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:29.746809 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:30.140991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:30.140953 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:30.141913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:30.141893 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:45:33.449600 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:45:33.449566 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:45:34.619843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.619793 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts"] Nov 02 03:45:34.624853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.624832 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.729759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.729718 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.729759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.729761 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.729946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.729836 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qj9t\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.729946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.729890 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.729946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.729910 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.730051 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.729957 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.730088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.730060 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.730126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.730088 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.730227 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.730206 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.731896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.731881 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f6e39-openshift-default-dockercfg-qc8xh\"" Nov 02 03:45:34.735496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.735478 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts"] Nov 02 03:45:34.831069 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831028 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831089 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831108 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831130 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qj9t\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831149 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831168 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831213 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831275 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831303 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831512 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831610 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831628 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831652 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.831962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.831939 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.833478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.833452 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.833851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.833833 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.890173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.890109 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qj9t\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.890292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.890275 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token\") pod \"e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:34.939710 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:34.939683 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:35.367165 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:35.367140 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3909121_92f8_4580_a841_2756399cfed5.slice/crio-0754c181de8ab117296de44bf0251b7da8104d979531c428fe4ebe605a2d3f6b WatchSource:0}: Error finding container 0754c181de8ab117296de44bf0251b7da8104d979531c428fe4ebe605a2d3f6b: Status 404 returned error can't find the container with id 0754c181de8ab117296de44bf0251b7da8104d979531c428fe4ebe605a2d3f6b Nov 02 03:45:35.367291 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:35.367267 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts"] Nov 02 03:45:35.369564 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:35.369533 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:35.369624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:35.369592 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:35.369624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:35.369620 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:36.207173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:36.207134 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" event={"ID":"c3909121-92f8-4580-a841-2756399cfed5","Type":"ContainerStarted","Data":"b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c"} Nov 02 03:45:36.208016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:36.207196 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" event={"ID":"c3909121-92f8-4580-a841-2756399cfed5","Type":"ContainerStarted","Data":"0754c181de8ab117296de44bf0251b7da8104d979531c428fe4ebe605a2d3f6b"} Nov 02 03:45:36.322471 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:36.322421 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podStartSLOduration=2.322404358 podStartE2EDuration="2.322404358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:36.320707626 +0000 UTC m=+3478.316408573" watchObservedRunningTime="2026-11-02 03:45:36.322404358 +0000 UTC m=+3478.318105300" Nov 02 03:45:36.950373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:36.943637 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:36.950373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:36.945829 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 03:45:36.950373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:36.945904 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 03:45:37.956376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:37.952676 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 03:45:37.956376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:37.952735 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 03:45:38.956502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:38.956465 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 03:45:38.956983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:38.956522 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 03:45:39.944572 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:39.944537 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 03:45:39.944855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:39.944827 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 03:45:40.946790 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:40.946752 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 03:45:40.947430 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:40.947397 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 03:45:41.942375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:41.940337 4868 patch_prober.go:29] interesting pod/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 02 03:45:41.942375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:41.940402 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 02 03:45:43.138227 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:43.138195 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:43.138717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:43.138696 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:43.139543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:43.139520 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:45:47.456899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.456867 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p"] Nov 02 03:45:47.461153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.461132 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471062 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471104 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471178 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471331 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471242 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471331 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471317 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471405 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjq48\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471440 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471456 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.471547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.471481 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573082 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573135 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573186 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573278 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573366 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjq48\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573405 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573429 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573518 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573472 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573564 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573634 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573682 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.573970 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.573951 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.574159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.574097 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.574512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.574489 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.577022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.576991 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.581889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.581869 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.630270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.630242 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjq48\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.633890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.633868 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token\") pod \"e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:47.658919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.658890 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p"] Nov 02 03:45:47.774827 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:47.774753 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:48.199060 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.199032 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p"] Nov 02 03:45:48.199453 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:45:48.199434 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bbd771_6e3c_4747_97de_8ae538409968.slice/crio-c6e77b9f4a6c24f4432db192f004d7a6a24e07f15464835366cbcb8d631295b3 WatchSource:0}: Error finding container c6e77b9f4a6c24f4432db192f004d7a6a24e07f15464835366cbcb8d631295b3: Status 404 returned error can't find the container with id c6e77b9f4a6c24f4432db192f004d7a6a24e07f15464835366cbcb8d631295b3 Nov 02 03:45:48.201924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.201893 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:48.202007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.201952 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:48.202007 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.201980 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:45:48.325859 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.325826 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" event={"ID":"40bbd771-6e3c-4747-97de-8ae538409968","Type":"ContainerStarted","Data":"ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902"} Nov 02 03:45:48.326027 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.325880 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" event={"ID":"40bbd771-6e3c-4747-97de-8ae538409968","Type":"ContainerStarted","Data":"c6e77b9f4a6c24f4432db192f004d7a6a24e07f15464835366cbcb8d631295b3"} Nov 02 03:45:48.442777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.442723 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podStartSLOduration=1.442708668 podStartE2EDuration="1.442708668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:45:48.441241746 +0000 UTC m=+3490.436942700" watchObservedRunningTime="2026-11-02 03:45:48.442708668 +0000 UTC m=+3490.438409611" Nov 02 03:45:48.775329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:48.775294 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:49.777062 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:49.777027 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 03:45:49.777455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:49.777094 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 03:45:50.775817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:50.775777 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 03:45:50.776024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:50.775835 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 03:45:51.775210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:51.775163 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 03:45:51.775633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:51.775235 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 03:45:52.775711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:52.775676 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 03:45:52.776109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:52.775728 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 03:45:53.775605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:53.775566 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 03:45:53.775770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:53.775622 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 03:45:54.775721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:54.775682 4868 patch_prober.go:29] interesting pod/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 03:45:54.775933 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:54.775759 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 03:45:55.779956 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:55.779928 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:55.780384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:55.780371 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:45:55.780922 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:45:55.780903 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:46:03.446997 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:46:03.446962 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:46:11.835455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:11.835422 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2"] Nov 02 03:46:11.839492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:11.839474 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:11.895496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:11.895466 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2"] Nov 02 03:46:11.940961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:11.940932 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:46:11.940961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:11.940936 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:46:12.021133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.021102 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.021273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.021164 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.021273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.021261 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8skq\" (UniqueName: \"kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.122265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.122233 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.122461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.122293 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.122461 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.122368 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8skq\" (UniqueName: \"kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.123011 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.122986 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.124627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.124595 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.178450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.178415 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8skq\" (UniqueName: \"kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq\") pod \"maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.453619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.453549 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:12.828404 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:46:12.828371 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2519a386_bb8e_4d1a_9c38_7410b5b6bf6f.slice/crio-14ba72a9048836ca8f8be0f9600aaeee21893d2e51bbf343f85b7f0ba6946fbc WatchSource:0}: Error finding container 14ba72a9048836ca8f8be0f9600aaeee21893d2e51bbf343f85b7f0ba6946fbc: Status 404 returned error can't find the container with id 14ba72a9048836ca8f8be0f9600aaeee21893d2e51bbf343f85b7f0ba6946fbc Nov 02 03:46:12.828838 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:12.828812 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2"] Nov 02 03:46:13.545040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:13.544953 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" event={"ID":"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f","Type":"ContainerStarted","Data":"d521333e1617488c4bf4cc0f154433dba82f7b2ea11f5c5170b021b657228d31"} Nov 02 03:46:13.545040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:13.545011 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" event={"ID":"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f","Type":"ContainerStarted","Data":"14ba72a9048836ca8f8be0f9600aaeee21893d2e51bbf343f85b7f0ba6946fbc"} Nov 02 03:46:13.657416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:13.657369 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" podStartSLOduration=2.657339029 podStartE2EDuration="2.657339029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:46:13.655505639 +0000 UTC m=+3515.651206583" watchObservedRunningTime="2026-11-02 03:46:13.657339029 +0000 UTC m=+3515.653039973" Nov 02 03:46:14.553905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:14.553870 4868 generic.go:356] "Generic (PLEG): container finished" podID="2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" containerID="d521333e1617488c4bf4cc0f154433dba82f7b2ea11f5c5170b021b657228d31" exitCode=0 Nov 02 03:46:14.554346 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:14.553956 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" event={"ID":"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f","Type":"ContainerDied","Data":"d521333e1617488c4bf4cc0f154433dba82f7b2ea11f5c5170b021b657228d31"} Nov 02 03:46:15.716476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.716451 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:15.763703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.763676 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq\" (UniqueName: \"kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq\") pod \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " Nov 02 03:46:15.763833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.763724 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca\") pod \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " Nov 02 03:46:15.763904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.763887 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca\") pod \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\" (UID: \"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f\") " Nov 02 03:46:15.764222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.764196 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca" pod "2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" (UID: "2519a386-bb8e-4d1a-9c38-7410b5b6bf6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:46:15.765708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.765680 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca" pod "2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" (UID: "2519a386-bb8e-4d1a-9c38-7410b5b6bf6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:46:15.765818 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.765734 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq" pod "2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" (UID: "2519a386-bb8e-4d1a-9c38-7410b5b6bf6f"). InnerVolumeSpecName "kube-api-access-m8skq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:46:15.865456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.865420 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8skq\" (UniqueName: \"kubernetes.io/projected/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-kube-api-access-m8skq\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:46:15.865595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.865457 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:46:15.865595 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:15.865473 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:46:16.569222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:16.569145 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" Nov 02 03:46:16.569222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:16.569179 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2" event={"ID":"2519a386-bb8e-4d1a-9c38-7410b5b6bf6f","Type":"ContainerDied","Data":"14ba72a9048836ca8f8be0f9600aaeee21893d2e51bbf343f85b7f0ba6946fbc"} Nov 02 03:46:16.569435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:16.569227 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ba72a9048836ca8f8be0f9600aaeee21893d2e51bbf343f85b7f0ba6946fbc" Nov 02 03:46:33.446417 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:46:33.446383 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:46:38.998787 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:46:38.998761 4868 scope.go:122] "RemoveContainer" containerID="d8fbdd3b46ba5e6aa5674dc605aa8201df66fa57057d64b4c16247f3cb232286" Nov 02 03:47:03.446513 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:47:03.446478 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:47:10.048098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.048060 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr"] Nov 02 03:47:10.097468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.097431 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407a3d33-xjztr"] Nov 02 03:47:10.114912 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.114879 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66eab69-0a4c-4779-89d3-8b2b20716b36" path="/var/lib/kubelet/pods/f66eab69-0a4c-4779-89d3-8b2b20716b36/volumes" Nov 02 03:47:10.886113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.886077 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5"] Nov 02 03:47:10.886599 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.886584 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" containerName="revoke-keys" Nov 02 03:47:10.886655 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.886601 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" containerName="revoke-keys" Nov 02 03:47:10.886695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.886690 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" containerName="revoke-keys" Nov 02 03:47:10.890344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.890324 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:10.942610 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.942586 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:47:10.945696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.945673 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5"] Nov 02 03:47:10.971783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.971750 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:10.971955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.971930 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tn9\" (UniqueName: \"kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:10.972021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.971986 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:10.990863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:10.990835 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:47:11.073672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.073642 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.074147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.073702 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tn9\" (UniqueName: \"kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.074147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.073733 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.074514 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.074492 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.076008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.075987 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.130019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.129995 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tn9\" (UniqueName: \"kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9\") pod \"maas-api-revoke-sub-6f885532-22qn5\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.204204 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.204144 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:11.583147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.583118 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5"] Nov 02 03:47:11.584048 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:47:11.584019 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3eec97_6aa4_482a_b10f_62b499481b38.slice/crio-fb801607f4ec87eaa383eb5e82c699cbceda2f6d8646596270921dec86af9355 WatchSource:0}: Error finding container fb801607f4ec87eaa383eb5e82c699cbceda2f6d8646596270921dec86af9355: Status 404 returned error can't find the container with id fb801607f4ec87eaa383eb5e82c699cbceda2f6d8646596270921dec86af9355 Nov 02 03:47:11.975099 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.975057 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" event={"ID":"8e3eec97-6aa4-482a-b10f-62b499481b38","Type":"ContainerStarted","Data":"59ab22d53890ca079f08b5e815ac23a7ae97efd1d2ec3704048ab3656d5825b2"} Nov 02 03:47:11.975273 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:11.975122 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" event={"ID":"8e3eec97-6aa4-482a-b10f-62b499481b38","Type":"ContainerStarted","Data":"fb801607f4ec87eaa383eb5e82c699cbceda2f6d8646596270921dec86af9355"} Nov 02 03:47:12.089594 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:12.089509 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" podStartSLOduration=2.089494967 podStartE2EDuration="2.089494967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:47:12.086420056 +0000 UTC m=+3574.082121009" watchObservedRunningTime="2026-11-02 03:47:12.089494967 +0000 UTC m=+3574.085195909" Nov 02 03:47:12.983970 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:12.983934 4868 generic.go:356] "Generic (PLEG): container finished" podID="8e3eec97-6aa4-482a-b10f-62b499481b38" containerID="59ab22d53890ca079f08b5e815ac23a7ae97efd1d2ec3704048ab3656d5825b2" exitCode=0 Nov 02 03:47:12.984171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:12.983986 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" event={"ID":"8e3eec97-6aa4-482a-b10f-62b499481b38","Type":"ContainerDied","Data":"59ab22d53890ca079f08b5e815ac23a7ae97efd1d2ec3704048ab3656d5825b2"} Nov 02 03:47:14.132453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.132432 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:14.224725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.224694 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9\" (UniqueName: \"kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9\") pod \"8e3eec97-6aa4-482a-b10f-62b499481b38\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " Nov 02 03:47:14.225022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.224989 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca\") pod \"8e3eec97-6aa4-482a-b10f-62b499481b38\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " Nov 02 03:47:14.225310 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.225291 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca\") pod \"8e3eec97-6aa4-482a-b10f-62b499481b38\" (UID: \"8e3eec97-6aa4-482a-b10f-62b499481b38\") " Nov 02 03:47:14.225735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.225695 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca" pod "8e3eec97-6aa4-482a-b10f-62b499481b38" (UID: "8e3eec97-6aa4-482a-b10f-62b499481b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:47:14.226255 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.226233 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:47:14.233952 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.233915 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca" pod "8e3eec97-6aa4-482a-b10f-62b499481b38" (UID: "8e3eec97-6aa4-482a-b10f-62b499481b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:47:14.234374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.234335 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9" pod "8e3eec97-6aa4-482a-b10f-62b499481b38" (UID: "8e3eec97-6aa4-482a-b10f-62b499481b38"). InnerVolumeSpecName "kube-api-access-p2tn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:47:14.327198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.327166 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tn9\" (UniqueName: \"kubernetes.io/projected/8e3eec97-6aa4-482a-b10f-62b499481b38-kube-api-access-p2tn9\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:47:14.327198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.327194 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3eec97-6aa4-482a-b10f-62b499481b38-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:47:14.999228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.999193 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" event={"ID":"8e3eec97-6aa4-482a-b10f-62b499481b38","Type":"ContainerDied","Data":"fb801607f4ec87eaa383eb5e82c699cbceda2f6d8646596270921dec86af9355"} Nov 02 03:47:14.999423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.999265 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb801607f4ec87eaa383eb5e82c699cbceda2f6d8646596270921dec86af9355" Nov 02 03:47:14.999423 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:14.999211 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5" Nov 02 03:47:28.047862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:28.047829 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg"] Nov 02 03:47:28.096652 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:28.096619 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0588b02-4rcdg"] Nov 02 03:47:28.116315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:28.116285 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465fd062-d546-453a-b343-95b93ae60c0d" path="/var/lib/kubelet/pods/465fd062-d546-453a-b343-95b93ae60c0d/volumes" Nov 02 03:47:33.473922 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:47:33.473878 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:47:38.561757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:38.561722 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:47:38.569773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:38.569748 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:47:39.022833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.022802 4868 scope.go:122] "RemoveContainer" containerID="692806991a25e23719379dd2807cb0cf35e240bb5358c3fba4cf36d0fef2aacb" Nov 02 03:47:39.033153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.033129 4868 scope.go:122] "RemoveContainer" containerID="9e699ab612ae60c72afa78cde0670db4db97f59b54305fdaf2b6ba655192ec04" Nov 02 03:47:39.043382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.043336 4868 scope.go:122] "RemoveContainer" containerID="5e3f2b6cc755e07cb0f9e6fa7fd175ad36d9ad3008904614696a3c4073300b9d" Nov 02 03:47:39.300584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.300507 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw"] Nov 02 03:47:39.302077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.301251 4868 cpu_manager.go:414] "containerMap: removing container" podUID="8e3eec97-6aa4-482a-b10f-62b499481b38" containerName="revoke-keys" Nov 02 03:47:39.302077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.301268 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3eec97-6aa4-482a-b10f-62b499481b38" containerName="revoke-keys" Nov 02 03:47:39.302077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.301422 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3eec97-6aa4-482a-b10f-62b499481b38" containerName="revoke-keys" Nov 02 03:47:39.304590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.304569 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.366389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.362536 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-ac7ab4-openshift-default-dockercfg-mxm6m\"" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404204 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404320 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404375 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404405 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdc29\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404450 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404482 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404557 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404598 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.405946 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.404621 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.474963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.474802 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw"] Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527033 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527127 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527163 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527188 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdc29\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527227 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527257 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527321 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527374 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527401 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.540463 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.527835 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.543128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.542196 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.543128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.542512 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.545115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.544814 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.545115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.545073 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.546879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.546854 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.564897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.564831 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.597458 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.597426 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.598241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.598215 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdc29\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29\") pod \"e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:39.620468 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:39.620445 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:40.127968 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.127937 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw"] Nov 02 03:47:40.129238 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:47:40.129211 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b1e1b5_0896_480f_8b77_e1d853f16f09.slice/crio-e335017ee155615e9862b8ce6bc4ef83f2aa7ea5aebc3239ff27e6d6de6af962 WatchSource:0}: Error finding container e335017ee155615e9862b8ce6bc4ef83f2aa7ea5aebc3239ff27e6d6de6af962: Status 404 returned error can't find the container with id e335017ee155615e9862b8ce6bc4ef83f2aa7ea5aebc3239ff27e6d6de6af962 Nov 02 03:47:40.131695 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.131663 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:40.131802 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.131746 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:40.131802 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.131776 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:40.254182 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.254149 4868 generic.go:356] "Generic (PLEG): container finished" podID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerID="2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398" exitCode=0 Nov 02 03:47:40.254369 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.254219 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" event={"ID":"968ffdd6-532a-4921-a7cc-b9a3a940580d","Type":"ContainerDied","Data":"2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398"} Nov 02 03:47:40.254716 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.254688 4868 scope.go:122] "RemoveContainer" containerID="2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398" Nov 02 03:47:40.256202 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.256174 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" event={"ID":"c7b1e1b5-0896-480f-8b77-e1d853f16f09","Type":"ContainerStarted","Data":"51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8"} Nov 02 03:47:40.256294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.256219 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" event={"ID":"c7b1e1b5-0896-480f-8b77-e1d853f16f09","Type":"ContainerStarted","Data":"e335017ee155615e9862b8ce6bc4ef83f2aa7ea5aebc3239ff27e6d6de6af962"} Nov 02 03:47:40.256405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.256382 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:40.256459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.256447 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:40.256494 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.256480 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:40.489292 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.489186 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podStartSLOduration=1.489170288 podStartE2EDuration="1.489170288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:47:40.486023062 +0000 UTC m=+3602.481724006" watchObservedRunningTime="2026-11-02 03:47:40.489170288 +0000 UTC m=+3602.484871296" Nov 02 03:47:40.624381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:40.623463 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:41.268848 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:41.268813 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" event={"ID":"968ffdd6-532a-4921-a7cc-b9a3a940580d","Type":"ContainerStarted","Data":"c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5"} Nov 02 03:47:41.387436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:41.387398 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:47:41.389396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:41.388695 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 03:47:41.389396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:41.388761 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 03:47:41.722393 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:41.721716 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:42.226846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:42.223992 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:47:42.293773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:42.293738 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:42.294547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:42.294529 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:47:42.381723 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:42.381684 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 03:47:42.381935 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:42.381850 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 03:47:43.382421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:43.381776 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 03:47:43.382421 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:43.381836 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 03:47:44.082525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:44.070597 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz"] Nov 02 03:47:44.115366 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:44.115324 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1b1b27a-8dclz"] Nov 02 03:47:44.381806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:44.381757 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 03:47:44.382029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:44.381818 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 03:47:45.387474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:45.384727 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 03:47:45.387474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:45.384803 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 03:47:46.116465 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:46.116424 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef561126-48d2-465b-b1b6-3e1a2ec0dc4d" path="/var/lib/kubelet/pods/ef561126-48d2-465b-b1b6-3e1a2ec0dc4d/volumes" Nov 02 03:47:46.383428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:46.381711 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 02 03:47:46.383428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:46.381798 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 02 03:47:47.403770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:47.403732 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:47:47.405400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:47.405372 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 03:47:48.905438 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:48.905398 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb"] Nov 02 03:47:48.910681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:48.910663 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:48.965801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:48.965766 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb"] Nov 02 03:47:48.977409 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:48.977134 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-ac7ab4\"" Nov 02 03:47:49.039812 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.039766 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-ac7ab4-dockercfg-v6kgc\"" Nov 02 03:47:49.044746 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.044713 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.045049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.045032 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5jd\" (UniqueName: \"kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd\") pod \"payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.125453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.125424 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp"] Nov 02 03:47:49.129937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.129912 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp"] Nov 02 03:47:49.130128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.130038 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.146162 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.146125 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.146294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.146229 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5jd\" (UniqueName: \"kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd\") pod \"payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.148335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.147042 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.204976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.204903 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5jd\" (UniqueName: \"kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd\") pod \"payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.226018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.225987 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:49.247141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.247112 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.247305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.247155 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8b6\" (UniqueName: \"kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6\") pod \"payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.348006 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.347959 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.348006 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.348008 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8b6\" (UniqueName: \"kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6\") pod \"payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.348881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.348854 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.404469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.404425 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8b6\" (UniqueName: \"kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6\") pod \"payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.452403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.452339 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:49.601079 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:47:49.601049 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a17e38b_9c40_4b3b_875c_791d4dfb6762.slice/crio-ac9aa5d6325c5d807f7c40dd8bc7e4f70bb3a063a1249ca394f159891a553904 WatchSource:0}: Error finding container ac9aa5d6325c5d807f7c40dd8bc7e4f70bb3a063a1249ca394f159891a553904: Status 404 returned error can't find the container with id ac9aa5d6325c5d807f7c40dd8bc7e4f70bb3a063a1249ca394f159891a553904 Nov 02 03:47:49.602271 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.602246 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb"] Nov 02 03:47:49.834361 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:49.834323 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp"] Nov 02 03:47:49.836115 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:47:49.836085 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea61930e_d4ea_460b_834d_49d0318bf194.slice/crio-e20709d12561bea2eb4ccc37331c78afb9730055b6373a5226579c02bab9b399 WatchSource:0}: Error finding container e20709d12561bea2eb4ccc37331c78afb9730055b6373a5226579c02bab9b399: Status 404 returned error can't find the container with id e20709d12561bea2eb4ccc37331c78afb9730055b6373a5226579c02bab9b399 Nov 02 03:47:50.364794 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.364759 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" event={"ID":"ea61930e-d4ea-460b-834d-49d0318bf194","Type":"ContainerStarted","Data":"1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb"} Nov 02 03:47:50.365250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.364817 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" event={"ID":"ea61930e-d4ea-460b-834d-49d0318bf194","Type":"ContainerStarted","Data":"e20709d12561bea2eb4ccc37331c78afb9730055b6373a5226579c02bab9b399"} Nov 02 03:47:50.365250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.364873 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:47:50.366701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.366672 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" event={"ID":"4a17e38b-9c40-4b3b-875c-791d4dfb6762","Type":"ContainerStarted","Data":"52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c"} Nov 02 03:47:50.366795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.366722 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" event={"ID":"4a17e38b-9c40-4b3b-875c-791d4dfb6762","Type":"ContainerStarted","Data":"ac9aa5d6325c5d807f7c40dd8bc7e4f70bb3a063a1249ca394f159891a553904"} Nov 02 03:47:50.366795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.366755 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:47:50.478323 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.478270 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" podStartSLOduration=1.478255975 podStartE2EDuration="1.478255975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:47:50.476568567 +0000 UTC m=+3612.472269540" watchObservedRunningTime="2026-11-02 03:47:50.478255975 +0000 UTC m=+3612.473956918" Nov 02 03:47:50.626340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:50.626244 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" podStartSLOduration=2.626227443 podStartE2EDuration="2.626227443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:47:50.598740438 +0000 UTC m=+3612.594441387" watchObservedRunningTime="2026-11-02 03:47:50.626227443 +0000 UTC m=+3612.621928385" Nov 02 03:47:51.374958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:51.374925 4868 generic.go:356] "Generic (PLEG): container finished" podID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerID="55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386" exitCode=0 Nov 02 03:47:51.375417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:51.374999 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" event={"ID":"bb801bbf-9cd2-489e-985f-cd17ef2638de","Type":"ContainerDied","Data":"55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386"} Nov 02 03:47:51.375514 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:51.375498 4868 scope.go:122] "RemoveContainer" containerID="55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386" Nov 02 03:47:51.378266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:51.378230 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:51.378398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:51.378301 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:51.378398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:51.378361 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:52.383307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:52.383273 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" event={"ID":"bb801bbf-9cd2-489e-985f-cd17ef2638de","Type":"ContainerStarted","Data":"d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a"} Nov 02 03:47:52.750431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:52.745708 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:47:52.750431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:52.745836 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 03:47:52.750431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:52.745880 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 03:47:53.751375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:53.750721 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 03:47:53.751375 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:53.750784 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 03:47:54.747382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:54.744682 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 03:47:54.747382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:54.744744 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 03:47:55.743721 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:55.743681 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 03:47:55.745083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:55.745051 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 03:47:56.717174 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.717139 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r"] Nov 02 03:47:56.721264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.721245 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.742373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.741672 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 03:47:56.742373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.741721 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 03:47:56.804095 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.804064 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r"] Nov 02 03:47:56.805086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.804994 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gp94\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805282 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805270 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805425 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805510 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805483 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805548 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805519 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805583 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805659 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805723 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805710 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.805769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.805737 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.874519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.874467 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3d14ee-openshift-default-dockercfg-qlcpc\"" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909057 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909142 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909218 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909276 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909311 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909390 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gp94\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909515 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909562 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.910179 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.909604 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.911431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.911406 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.912832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.912805 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.913239 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.913216 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.914303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.914279 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.914743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.914711 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.914898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.914870 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.935990 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.935952 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.978908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.978834 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gp94\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:56.979558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:56.979528 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token\") pod \"e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:57.040376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.039552 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:57.551121 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:47:57.551082 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01c3224_7761_4203_858f_c7934ed114fb.slice/crio-a6dcc89dc0ec0efa80eb90f0eaff7a975ea8291053eda6e1a41fbcd26d44405f WatchSource:0}: Error finding container a6dcc89dc0ec0efa80eb90f0eaff7a975ea8291053eda6e1a41fbcd26d44405f: Status 404 returned error can't find the container with id a6dcc89dc0ec0efa80eb90f0eaff7a975ea8291053eda6e1a41fbcd26d44405f Nov 02 03:47:57.551897 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.551873 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r"] Nov 02 03:47:57.554435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.554406 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:57.554543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.554479 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:57.554543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.554530 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:47:57.585523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.585493 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" event={"ID":"a01c3224-7761-4203-858f-c7934ed114fb","Type":"ContainerStarted","Data":"a6dcc89dc0ec0efa80eb90f0eaff7a975ea8291053eda6e1a41fbcd26d44405f"} Nov 02 03:47:57.758926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.758897 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:47:57.759413 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.759386 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:47:57.760848 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:57.760817 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 03:47:58.045727 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:58.045689 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4"] Nov 02 03:47:58.048785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:58.048763 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a68e352d-zlwn4"] Nov 02 03:47:58.118251 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:58.118220 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f91ed18-412b-4c71-b694-ae9e0671ca2e" path="/var/lib/kubelet/pods/9f91ed18-412b-4c71-b694-ae9e0671ca2e/volumes" Nov 02 03:47:58.602333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:58.602288 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" event={"ID":"a01c3224-7761-4203-858f-c7934ed114fb","Type":"ContainerStarted","Data":"663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689"} Nov 02 03:47:58.727512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:58.727458 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podStartSLOduration=2.727443138 podStartE2EDuration="2.727443138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:47:58.71672225 +0000 UTC m=+3620.712423194" watchObservedRunningTime="2026-11-02 03:47:58.727443138 +0000 UTC m=+3620.723144080" Nov 02 03:47:59.042574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:59.042493 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:59.047964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:59.047941 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:59.610019 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:59.609987 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:47:59.610803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:47:59.610783 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:48:01.045411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:01.045377 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v"] Nov 02 03:48:01.094853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:01.094824 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de7748e3-pmt8v"] Nov 02 03:48:01.377813 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:01.377780 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 03:48:01.377993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:01.377859 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 03:48:02.116151 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:02.116120 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafa5e25-bc39-4f3b-ac32-9c329e2f98a6" path="/var/lib/kubelet/pods/cafa5e25-bc39-4f3b-ac32-9c329e2f98a6/volumes" Nov 02 03:48:03.446575 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:48:03.446543 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:48:05.234338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.234296 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv"] Nov 02 03:48:05.280872 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.280831 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.313688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.313653 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv"] Nov 02 03:48:05.382665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.382635 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3d14ee-dockercfg-75r8p\"" Nov 02 03:48:05.382665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.382663 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3d14ee\"" Nov 02 03:48:05.472809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.472779 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvnx\" (UniqueName: \"kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx\") pod \"payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.473005 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.472951 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.574083 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.574010 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.574221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.574113 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvnx\" (UniqueName: \"kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx\") pod \"payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.574759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.574739 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.630776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.630748 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvnx\" (UniqueName: \"kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx\") pod \"payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:05.923994 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:05.923962 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:06.299822 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:06.299795 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv"] Nov 02 03:48:06.300240 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:48:06.299876 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6422fe_c398_4865_8bf2_b83df32106d0.slice/crio-14f4b4d90baecb68a835697cf3d9c630b7ad3aba998249f241053b8f8ef0a0b3 WatchSource:0}: Error finding container 14f4b4d90baecb68a835697cf3d9c630b7ad3aba998249f241053b8f8ef0a0b3: Status 404 returned error can't find the container with id 14f4b4d90baecb68a835697cf3d9c630b7ad3aba998249f241053b8f8ef0a0b3 Nov 02 03:48:06.662150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:06.662115 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" event={"ID":"cd6422fe-c398-4865-8bf2-b83df32106d0","Type":"ContainerStarted","Data":"25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d"} Nov 02 03:48:06.662330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:06.662165 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" event={"ID":"cd6422fe-c398-4865-8bf2-b83df32106d0","Type":"ContainerStarted","Data":"14f4b4d90baecb68a835697cf3d9c630b7ad3aba998249f241053b8f8ef0a0b3"} Nov 02 03:48:06.662330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:06.662242 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:10.044954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:10.044893 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" podStartSLOduration=5.044879365 podStartE2EDuration="5.044879365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:48:06.803267253 +0000 UTC m=+3628.798968196" watchObservedRunningTime="2026-11-02 03:48:10.044879365 +0000 UTC m=+3632.040580310" Nov 02 03:48:10.046476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:10.046449 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27"] Nov 02 03:48:10.095340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:10.095308 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e5a37-hdh27"] Nov 02 03:48:10.115530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:10.115497 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d1404a-7697-4309-ae4e-edddf5d602a1" path="/var/lib/kubelet/pods/67d1404a-7697-4309-ae4e-edddf5d602a1/volumes" Nov 02 03:48:10.678191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:10.678159 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh"] Nov 02 03:48:10.678389 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:10.678338 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" containerID="cri-o://56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2" gracePeriod=30 Nov 02 03:48:13.050226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:13.050197 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl"] Nov 02 03:48:13.099455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:13.099421 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daae74a4-kmgtl"] Nov 02 03:48:14.116316 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:14.116287 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec43308-3716-4db1-a8ac-5b3b30f041bd" path="/var/lib/kubelet/pods/fec43308-3716-4db1-a8ac-5b3b30f041bd/volumes" Nov 02 03:48:15.936247 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:15.936223 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:48:16.012104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012024 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012076 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012104 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012098 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012217 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012384 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012387 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:48:16.012531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012485 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012503 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:48:16.012658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012548 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012568 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:48:16.012658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012578 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012621 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo\") pod \"23b7191e-00d3-4cae-8883-2ca020f5467b\" (UID: \"23b7191e-00d3-4cae-8883-2ca020f5467b\") " Nov 02 03:48:16.012843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012754 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:48:16.012960 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.012939 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:48:16.013098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.013080 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.013196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.013099 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.013196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.013108 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.013196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.013117 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b7191e-00d3-4cae-8883-2ca020f5467b-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.013196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.013125 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.014486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.014460 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:48:16.014709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.014686 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:48:16.014857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.014838 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "kube-api-access-rknks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:48:16.014920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.014848 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token" pod "23b7191e-00d3-4cae-8883-2ca020f5467b" (UID: "23b7191e-00d3-4cae-8883-2ca020f5467b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:48:16.114197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.114103 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.114197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.114140 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.114197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.114155 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b7191e-00d3-4cae-8883-2ca020f5467b-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.114197 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.114170 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rknks\" (UniqueName: \"kubernetes.io/projected/23b7191e-00d3-4cae-8883-2ca020f5467b-kube-api-access-rknks\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:48:16.744191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.744160 4868 generic.go:356] "Generic (PLEG): container finished" podID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerID="56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2" exitCode=0 Nov 02 03:48:16.744416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.744206 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" Nov 02 03:48:16.744416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.744242 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" event={"ID":"23b7191e-00d3-4cae-8883-2ca020f5467b","Type":"ContainerDied","Data":"56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2"} Nov 02 03:48:16.744416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.744303 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh" event={"ID":"23b7191e-00d3-4cae-8883-2ca020f5467b","Type":"ContainerDied","Data":"7da1dae53d401ee616e3df76ed36f609c6986a01c15dd0b1b3a8a715ff38d609"} Nov 02 03:48:16.744416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.744332 4868 scope.go:122] "RemoveContainer" containerID="56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2" Nov 02 03:48:16.754338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.754322 4868 scope.go:122] "RemoveContainer" containerID="730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae" Nov 02 03:48:16.763704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.763688 4868 scope.go:122] "RemoveContainer" containerID="56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2" Nov 02 03:48:16.763968 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:48:16.763948 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2\": container with ID starting with 56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2 not found: ID does not exist" containerID="56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2" Nov 02 03:48:16.764038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.763976 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2"} err="failed to get container status \"56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2\": rpc error: code = NotFound desc = could not find container \"56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2\": container with ID starting with 56c0addbe80809dd4989bad833502b25ba759cc9d2873c5af4abb3f7166a14f2 not found: ID does not exist" Nov 02 03:48:16.764038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.763994 4868 scope.go:122] "RemoveContainer" containerID="730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae" Nov 02 03:48:16.764241 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:48:16.764220 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae\": container with ID starting with 730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae not found: ID does not exist" containerID="730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae" Nov 02 03:48:16.764287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.764245 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae"} err="failed to get container status \"730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae\": rpc error: code = NotFound desc = could not find container \"730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae\": container with ID starting with 730619792f5c4f599fce6154453d1a2b392cd2af29f5e123819c9076b9c7baae not found: ID does not exist" Nov 02 03:48:16.908850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.908821 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh"] Nov 02 03:48:16.958915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:16.958890 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f68c7e-openshift-default-db7d68949-slfvh"] Nov 02 03:48:17.671815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:17.671788 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 03:48:18.122737 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:18.122705 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" path="/var/lib/kubelet/pods/23b7191e-00d3-4cae-8883-2ca020f5467b/volumes" Nov 02 03:48:24.820500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:24.820417 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2"] Nov 02 03:48:24.869999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:24.869970 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1b1008-dtqk2"] Nov 02 03:48:26.049098 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:26.049065 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975"] Nov 02 03:48:26.098543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:26.098511 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8eb801-86975"] Nov 02 03:48:26.116329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:26.116296 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106e6e43-0ec0-4353-9212-2596c81cefe3" path="/var/lib/kubelet/pods/106e6e43-0ec0-4353-9212-2596c81cefe3/volumes" Nov 02 03:48:26.116902 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:26.116886 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2519a386-bb8e-4d1a-9c38-7410b5b6bf6f" path="/var/lib/kubelet/pods/2519a386-bb8e-4d1a-9c38-7410b5b6bf6f/volumes" Nov 02 03:48:28.045991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:28.045962 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l"] Nov 02 03:48:28.052803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:28.052777 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec06e73e-phn7l"] Nov 02 03:48:28.115342 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:28.115316 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a072928c-e8d8-425d-a3ad-9efe9db46e27" path="/var/lib/kubelet/pods/a072928c-e8d8-425d-a3ad-9efe9db46e27/volumes" Nov 02 03:48:33.447029 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:48:33.446995 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:48:39.083522 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:39.083487 4868 scope.go:122] "RemoveContainer" containerID="118bfada98a6dfea68149c123fba270a8ebd8a5dc96ec6097af5c023acbfc6df" Nov 02 03:48:39.093927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:39.093904 4868 scope.go:122] "RemoveContainer" containerID="526410c6af912dd685d8b4b9a90ee42675ac8bca1335469d5c2a2b92cc214c2a" Nov 02 03:48:39.102978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:39.102957 4868 scope.go:122] "RemoveContainer" containerID="b6a9150773faef9845bb782a2502aa11aa12b728ae8be5e842a758b4e084c0fd" Nov 02 03:48:45.046270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:45.046227 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf"] Nov 02 03:48:45.094706 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:45.094674 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afd209fa-nkglf"] Nov 02 03:48:46.115185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:46.115152 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87" path="/var/lib/kubelet/pods/bb7e65f0-4cc4-4a37-9f5f-0e61e7bddd87/volumes" Nov 02 03:48:52.046386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:52.046344 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk"] Nov 02 03:48:52.095647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:52.095615 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521a73dc-nzsbk"] Nov 02 03:48:52.114785 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:48:52.114757 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf38542e-ce0d-46e6-8511-34dc80dce119" path="/var/lib/kubelet/pods/cf38542e-ce0d-46e6-8511-34dc80dce119/volumes" Nov 02 03:49:03.446478 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:49:03.446434 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:49:11.046018 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:11.045985 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s"] Nov 02 03:49:11.049097 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:11.049069 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ede3cdd-sb89s"] Nov 02 03:49:12.116270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:12.116230 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc02105-b67e-4857-9ed9-6610af43d378" path="/var/lib/kubelet/pods/cbc02105-b67e-4857-9ed9-6610af43d378/volumes" Nov 02 03:49:13.325517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:13.325486 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc"] Nov 02 03:49:13.325958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:13.325707 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" containerID="cri-o://b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14" gracePeriod=30 Nov 02 03:49:14.117367 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117335 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449"] Nov 02 03:49:14.117852 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117838 4868 cpu_manager.go:414] "containerMap: removing container" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" Nov 02 03:49:14.117915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117853 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" Nov 02 03:49:14.117915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117881 4868 cpu_manager.go:414] "containerMap: removing container" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" Nov 02 03:49:14.117915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117886 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" Nov 02 03:49:14.118022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117970 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" Nov 02 03:49:14.118022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.117983 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b7191e-00d3-4cae-8883-2ca020f5467b" containerName="istio-proxy" Nov 02 03:49:14.121580 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.121562 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.168674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.168648 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449"] Nov 02 03:49:14.173235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.173207 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:49:14.173403 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.173335 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:49:14.259860 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.259822 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.260039 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.259988 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9trg\" (UniqueName: \"kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.260138 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.260112 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.361669 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.361635 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.362072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.361709 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9trg\" (UniqueName: \"kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.362072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.361762 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.362317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.362291 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.364090 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.364069 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.417807 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.417744 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9trg\" (UniqueName: \"kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg\") pod \"maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.434816 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.434788 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:14.812636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:14.812607 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449"] Nov 02 03:49:14.813498 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:49:14.813474 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4711c067_4ebc_483f_8dbb_8aeaf94a7a9e.slice/crio-ff4e0d87adda2bf6b064a4a6cf726379936e1ae5b810c55a0acd292b9e025d5d WatchSource:0}: Error finding container ff4e0d87adda2bf6b064a4a6cf726379936e1ae5b810c55a0acd292b9e025d5d: Status 404 returned error can't find the container with id ff4e0d87adda2bf6b064a4a6cf726379936e1ae5b810c55a0acd292b9e025d5d Nov 02 03:49:15.166217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:15.166185 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" event={"ID":"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e","Type":"ContainerStarted","Data":"887faa2d3d50df6c24b2b430e075055397d0a3ca1b74a3ea28c39c53b331c574"} Nov 02 03:49:15.166368 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:15.166250 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" event={"ID":"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e","Type":"ContainerStarted","Data":"ff4e0d87adda2bf6b064a4a6cf726379936e1ae5b810c55a0acd292b9e025d5d"} Nov 02 03:49:15.288748 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:15.288686 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" podStartSLOduration=1.288524206 podStartE2EDuration="1.288524206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:49:15.277593396 +0000 UTC m=+3697.273294338" watchObservedRunningTime="2026-11-02 03:49:15.288524206 +0000 UTC m=+3697.284225148" Nov 02 03:49:16.175617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:16.175584 4868 generic.go:356] "Generic (PLEG): container finished" podID="4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" containerID="887faa2d3d50df6c24b2b430e075055397d0a3ca1b74a3ea28c39c53b331c574" exitCode=0 Nov 02 03:49:16.175617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:16.175622 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" event={"ID":"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e","Type":"ContainerDied","Data":"887faa2d3d50df6c24b2b430e075055397d0a3ca1b74a3ea28c39c53b331c574"} Nov 02 03:49:17.339527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.339502 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:17.496163 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.496090 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg\" (UniqueName: \"kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg\") pod \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " Nov 02 03:49:17.496312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.496187 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca\") pod \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " Nov 02 03:49:17.496312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.496253 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca\") pod \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\" (UID: \"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e\") " Nov 02 03:49:17.496573 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.496543 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca" pod "4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" (UID: "4711c067-4ebc-483f-8dbb-8aeaf94a7a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:49:17.496724 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.496706 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:17.498284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.498265 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca" pod "4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" (UID: "4711c067-4ebc-483f-8dbb-8aeaf94a7a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:49:17.498401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.498337 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg" pod "4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" (UID: "4711c067-4ebc-483f-8dbb-8aeaf94a7a9e"). InnerVolumeSpecName "kube-api-access-m9trg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:49:17.597425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.597398 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:17.597425 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:17.597422 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9trg\" (UniqueName: \"kubernetes.io/projected/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e-kube-api-access-m9trg\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.192553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.192516 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" Nov 02 03:49:18.192732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.192518 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449" event={"ID":"4711c067-4ebc-483f-8dbb-8aeaf94a7a9e","Type":"ContainerDied","Data":"ff4e0d87adda2bf6b064a4a6cf726379936e1ae5b810c55a0acd292b9e025d5d"} Nov 02 03:49:18.192732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.192645 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff4e0d87adda2bf6b064a4a6cf726379936e1ae5b810c55a0acd292b9e025d5d" Nov 02 03:49:18.593469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.593446 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:49:18.708659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708583 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.708659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708636 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.708858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708665 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.708858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708743 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.708858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708797 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.709043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708873 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.709043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708896 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:49:18.709043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708940 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.709043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.708990 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.709043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709025 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data\") pod \"dc327185-945b-4338-ae87-1c38841a752f\" (UID: \"dc327185-945b-4338-ae87-1c38841a752f\") " Nov 02 03:49:18.709312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709107 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:49:18.709312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709197 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:49:18.709312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709227 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:49:18.709512 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709392 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:49:18.709665 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709648 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc327185-945b-4338-ae87-1c38841a752f-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.709752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709671 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.709752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709684 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.709752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709697 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.709752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.709711 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.711088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.711051 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:49:18.711217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.711094 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:49:18.711312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.711296 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:49:18.711438 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.711423 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr" pod "dc327185-945b-4338-ae87-1c38841a752f" (UID: "dc327185-945b-4338-ae87-1c38841a752f"). InnerVolumeSpecName "kube-api-access-bzvpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:49:18.812702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.811675 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.812702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.811715 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvpr\" (UniqueName: \"kubernetes.io/projected/dc327185-945b-4338-ae87-1c38841a752f-kube-api-access-bzvpr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.812702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.811732 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc327185-945b-4338-ae87-1c38841a752f-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:18.812702 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:18.811746 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc327185-945b-4338-ae87-1c38841a752f-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:19.201708 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.201672 4868 generic.go:356] "Generic (PLEG): container finished" podID="dc327185-945b-4338-ae87-1c38841a752f" containerID="b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14" exitCode=0 Nov 02 03:49:19.201894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.201720 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" Nov 02 03:49:19.201894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.201760 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" event={"ID":"dc327185-945b-4338-ae87-1c38841a752f","Type":"ContainerDied","Data":"b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14"} Nov 02 03:49:19.201894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.201816 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc" event={"ID":"dc327185-945b-4338-ae87-1c38841a752f","Type":"ContainerDied","Data":"0c7318a290d1be87c84f15d704f5f6ec65e89478a9d9c6290200773a63261425"} Nov 02 03:49:19.201894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.201838 4868 scope.go:122] "RemoveContainer" containerID="b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14" Nov 02 03:49:19.212991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.212965 4868 scope.go:122] "RemoveContainer" containerID="8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893" Nov 02 03:49:19.222941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.222924 4868 scope.go:122] "RemoveContainer" containerID="b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14" Nov 02 03:49:19.223233 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:49:19.223214 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14\": container with ID starting with b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14 not found: ID does not exist" containerID="b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14" Nov 02 03:49:19.223298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.223241 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14"} err="failed to get container status \"b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14\": rpc error: code = NotFound desc = could not find container \"b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14\": container with ID starting with b7b54d7ccba2edd323d1ca5df012c755581e99a2ca0105f6cf5284c37e672f14 not found: ID does not exist" Nov 02 03:49:19.223298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.223261 4868 scope.go:122] "RemoveContainer" containerID="8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893" Nov 02 03:49:19.223547 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:49:19.223523 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893\": container with ID starting with 8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893 not found: ID does not exist" containerID="8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893" Nov 02 03:49:19.223603 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.223555 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893"} err="failed to get container status \"8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893\": rpc error: code = NotFound desc = could not find container \"8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893\": container with ID starting with 8a8a16c6df5de70cb12870a0aabb811f625283181877b19a00431ebbcd27d893 not found: ID does not exist" Nov 02 03:49:19.364816 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.364780 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc"] Nov 02 03:49:19.414147 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:19.414114 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1008-openshift-default-678bcc8777-8l6rc"] Nov 02 03:49:20.122413 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:20.122383 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc327185-945b-4338-ae87-1c38841a752f" path="/var/lib/kubelet/pods/dc327185-945b-4338-ae87-1c38841a752f/volumes" Nov 02 03:49:25.460955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:25.460902 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs"] Nov 02 03:49:26.057290 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:26.057197 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj"] Nov 02 03:49:26.060478 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:26.060448 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcea670-cnlrj"] Nov 02 03:49:26.115985 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:26.115952 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d94b01f-307b-4d89-987c-18cadad403c7" path="/var/lib/kubelet/pods/2d94b01f-307b-4d89-987c-18cadad403c7/volumes" Nov 02 03:49:30.462082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.462036 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" podUID="528bbd62-8941-4d37-84f8-e8e8618bc8bd" containerName="payload-processing" containerID="cri-o://f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63" gracePeriod=25 Nov 02 03:49:30.737190 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.737163 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:49:30.776235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.776203 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6\" (UniqueName: \"kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6\") pod \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " Nov 02 03:49:30.776370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.776266 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume\") pod \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\" (UID: \"528bbd62-8941-4d37-84f8-e8e8618bc8bd\") " Nov 02 03:49:30.776666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.776645 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume" pod "528bbd62-8941-4d37-84f8-e8e8618bc8bd" (UID: "528bbd62-8941-4d37-84f8-e8e8618bc8bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:49:30.776777 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.776763 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528bbd62-8941-4d37-84f8-e8e8618bc8bd-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:30.778300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.778280 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6" pod "528bbd62-8941-4d37-84f8-e8e8618bc8bd" (UID: "528bbd62-8941-4d37-84f8-e8e8618bc8bd"). InnerVolumeSpecName "kube-api-access-vw9w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:49:30.878258 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:30.878217 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9w6\" (UniqueName: \"kubernetes.io/projected/528bbd62-8941-4d37-84f8-e8e8618bc8bd-kube-api-access-vw9w6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:49:31.325749 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.325702 4868 generic.go:356] "Generic (PLEG): container finished" podID="528bbd62-8941-4d37-84f8-e8e8618bc8bd" containerID="f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63" exitCode=0 Nov 02 03:49:31.325943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.325764 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" Nov 02 03:49:31.325943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.325778 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" event={"ID":"528bbd62-8941-4d37-84f8-e8e8618bc8bd","Type":"ContainerDied","Data":"f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63"} Nov 02 03:49:31.325943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.325828 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs" event={"ID":"528bbd62-8941-4d37-84f8-e8e8618bc8bd","Type":"ContainerDied","Data":"a828dcc29a10d9ebbbdec4438d0dbbd0905484506c278c8287d978f0b92bd2a2"} Nov 02 03:49:31.325943 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.325849 4868 scope.go:122] "RemoveContainer" containerID="f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63" Nov 02 03:49:31.339881 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.339851 4868 scope.go:122] "RemoveContainer" containerID="f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63" Nov 02 03:49:31.340440 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:49:31.340409 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63\": container with ID starting with f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63 not found: ID does not exist" containerID="f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63" Nov 02 03:49:31.340581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.340446 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63"} err="failed to get container status \"f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63\": rpc error: code = NotFound desc = could not find container \"f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63\": container with ID starting with f2d306215c88707f080a5d4119fb0339681ee043701720124d89e0032460ab63 not found: ID does not exist" Nov 02 03:49:31.492169 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.492128 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs"] Nov 02 03:49:31.539422 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:31.539388 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f68c7e-575598759-2qcfs"] Nov 02 03:49:32.045563 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:32.045525 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg"] Nov 02 03:49:32.094652 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:32.094615 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db3591af-j2vfg"] Nov 02 03:49:32.115490 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:32.115459 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528bbd62-8941-4d37-84f8-e8e8618bc8bd" path="/var/lib/kubelet/pods/528bbd62-8941-4d37-84f8-e8e8618bc8bd/volumes" Nov 02 03:49:32.115938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:32.115919 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6" path="/var/lib/kubelet/pods/6ab7ecb8-8f2b-4654-82d8-86dcd1d1d1a6/volumes" Nov 02 03:49:33.446182 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:49:33.446142 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:49:34.350112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:34.350080 4868 generic.go:356] "Generic (PLEG): container finished" podID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerID="51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8" exitCode=0 Nov 02 03:49:34.350328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:34.350158 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" event={"ID":"c7b1e1b5-0896-480f-8b77-e1d853f16f09","Type":"ContainerDied","Data":"51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8"} Nov 02 03:49:34.350656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:34.350638 4868 scope.go:122] "RemoveContainer" containerID="51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8" Nov 02 03:49:34.352466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:34.352440 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:49:34.352558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:34.352509 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:49:34.352558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:34.352541 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:49:35.365513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:35.365477 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" event={"ID":"c7b1e1b5-0896-480f-8b77-e1d853f16f09","Type":"ContainerStarted","Data":"33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15"} Nov 02 03:49:35.621123 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:35.621049 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:49:35.621328 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:35.621306 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 03:49:35.621400 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:35.621381 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 03:49:36.621659 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:36.621624 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 03:49:36.622072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:36.621676 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 03:49:37.621477 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:37.621442 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 03:49:37.621688 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:37.621501 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 03:49:38.621471 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:38.621435 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 03:49:38.621664 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:38.621518 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 03:49:39.154374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.153415 4868 scope.go:122] "RemoveContainer" containerID="b031c21ae6953adc651b350e8cb1a9b674f1a9fe9ba925d88d835778e45d8405" Nov 02 03:49:39.165701 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.165679 4868 scope.go:122] "RemoveContainer" containerID="16595fe2438b5e31d0eb0748ee09ca2ca29c3d7c3ba2636b6cf07024e9c449ff" Nov 02 03:49:39.174426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.174401 4868 scope.go:122] "RemoveContainer" containerID="c8489261cb0d8075de53092f5800411098459d68b40bed024b24174cb9be0381" Nov 02 03:49:39.182884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.182865 4868 scope.go:122] "RemoveContainer" containerID="8d0fd5cd42f4913b6a09c16a822b1744a0d1c22168a72e78d563670da55bf635" Nov 02 03:49:39.192392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.192371 4868 scope.go:122] "RemoveContainer" containerID="083520517c2501c6431ccb1d7be30802565d59ab5ca02bab58ecb2f7e8dee919" Nov 02 03:49:39.200618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.200599 4868 scope.go:122] "RemoveContainer" containerID="d4757c40788ea9e9616a93c9270d825b09dc15de68623cc6a86646a9043ab5ed" Nov 02 03:49:39.208971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.208955 4868 scope.go:122] "RemoveContainer" containerID="3f8ad68350c70689bed2ed2b71cd1582e08b200b768dae3afa01d3734996d080" Nov 02 03:49:39.217318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.217300 4868 scope.go:122] "RemoveContainer" containerID="e00a9a2cfb2032848f55a94bea42ecd52cfbcf76c6bcc04197a17d6e785b1ad8" Nov 02 03:49:39.226073 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.226056 4868 scope.go:122] "RemoveContainer" containerID="ff0ea610d3aa52a614163194e89111dd1f5b00f28b7eb95d52faa0ca804d9dd5" Nov 02 03:49:39.620778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.620746 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 03:49:39.620966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:39.620805 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 03:49:40.625899 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:40.625869 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:49:40.626368 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:40.626309 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:49:40.627118 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:40.627095 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 03:49:42.046391 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:42.046340 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j"] Nov 02 03:49:42.095648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:42.095615 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb1515ee-mwk9j"] Nov 02 03:49:42.115709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:42.115678 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4732e0c-3043-4985-ab02-78bb7b7060b2" path="/var/lib/kubelet/pods/b4732e0c-3043-4985-ab02-78bb7b7060b2/volumes" Nov 02 03:49:48.045333 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:48.045299 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7"] Nov 02 03:49:48.174234 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:48.174207 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7"] Nov 02 03:49:48.179734 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:49:48.179683 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e18790f4-fslq7\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e18790f4-fslq7" Nov 02 03:49:49.277768 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:49.277680 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449"] Nov 02 03:49:49.326788 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:49.326757 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f68c7e-5l449"] Nov 02 03:49:50.046636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:50.046598 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s"] Nov 02 03:49:50.095401 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:50.095372 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870276d-6m95s"] Nov 02 03:49:50.115309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:50.115272 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ecae1e-6d09-48d9-b01b-2d18898e5364" path="/var/lib/kubelet/pods/37ecae1e-6d09-48d9-b01b-2d18898e5364/volumes" Nov 02 03:49:50.115896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:50.115876 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" path="/var/lib/kubelet/pods/4711c067-4ebc-483f-8dbb-8aeaf94a7a9e/volumes" Nov 02 03:49:50.116442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:50.116419 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4e90fd-1ab5-41c6-bfbc-e23629a7167b" path="/var/lib/kubelet/pods/9b4e90fd-1ab5-41c6-bfbc-e23629a7167b/volumes" Nov 02 03:49:57.406415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.406379 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln"] Nov 02 03:49:57.407101 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407078 4868 cpu_manager.go:414] "containerMap: removing container" podUID="528bbd62-8941-4d37-84f8-e8e8618bc8bd" containerName="payload-processing" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407103 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528bbd62-8941-4d37-84f8-e8e8618bc8bd" containerName="payload-processing" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407122 4868 cpu_manager.go:414] "containerMap: removing container" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407129 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407148 4868 cpu_manager.go:414] "containerMap: removing container" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407156 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407185 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" containerName="revoke-keys" Nov 02 03:49:57.407207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407192 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" containerName="revoke-keys" Nov 02 03:49:57.407649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407342 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="528bbd62-8941-4d37-84f8-e8e8618bc8bd" containerName="payload-processing" Nov 02 03:49:57.407649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407374 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" Nov 02 03:49:57.407649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.407392 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4711c067-4ebc-483f-8dbb-8aeaf94a7a9e" containerName="revoke-keys" Nov 02 03:49:57.412056 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.412033 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464471 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464566 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464612 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464662 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464698 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464765 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464860 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.464961 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.467374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.465003 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884zq\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.540553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.540520 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-86f4c4-openshift-default-dockercfg-mngrr\"" Nov 02 03:49:57.549871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.549781 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln"] Nov 02 03:49:57.575175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.575128 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.585945 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.585916 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884zq\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.590507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.590441 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.602575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.602542 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.603113 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.603063 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.604046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.604027 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.604975 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.604514 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.622742 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.622706 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.653438 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.653403 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.655232 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.603949 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.659511 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.651733 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.659682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.604449 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.659813 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.654673 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.659913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.584443 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.732434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.726613 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-884zq\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.732434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.727977 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.732434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.731085 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:57.732434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:57.731417 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo\") pod \"e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:58.028798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:58.028705 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:49:58.490205 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:58.490171 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln"] Nov 02 03:49:58.491467 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:49:58.491434 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5c9167_8e93_4fdc_871d_2d160226364d.slice/crio-78aba9fa38d3ed41d6959e8e4cfb1f105cfca924f35dc762bcceea8844cf7b75 WatchSource:0}: Error finding container 78aba9fa38d3ed41d6959e8e4cfb1f105cfca924f35dc762bcceea8844cf7b75: Status 404 returned error can't find the container with id 78aba9fa38d3ed41d6959e8e4cfb1f105cfca924f35dc762bcceea8844cf7b75 Nov 02 03:49:58.494211 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:58.494172 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:49:58.494334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:58.494240 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:49:58.494334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:58.494269 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:49:58.591991 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:58.591963 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" event={"ID":"7d5c9167-8e93-4fdc-871d-2d160226364d","Type":"ContainerStarted","Data":"78aba9fa38d3ed41d6959e8e4cfb1f105cfca924f35dc762bcceea8844cf7b75"} Nov 02 03:49:59.600226 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:59.600185 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" event={"ID":"7d5c9167-8e93-4fdc-871d-2d160226364d","Type":"ContainerStarted","Data":"0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc"} Nov 02 03:49:59.769249 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:49:59.769190 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podStartSLOduration=2.769177252 podStartE2EDuration="2.769177252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:49:59.767841858 +0000 UTC m=+3741.763542798" watchObservedRunningTime="2026-11-02 03:49:59.769177252 +0000 UTC m=+3741.764878270" Nov 02 03:50:00.029440 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:00.029405 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:50:00.052377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:00.051628 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:50:00.607662 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:00.607631 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:50:00.608696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:00.608677 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:50:03.446202 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:50:03.446160 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:50:04.636225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:04.636189 4868 generic.go:356] "Generic (PLEG): container finished" podID="a01c3224-7761-4203-858f-c7934ed114fb" containerID="663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689" exitCode=0 Nov 02 03:50:04.636784 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:04.636262 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" event={"ID":"a01c3224-7761-4203-858f-c7934ed114fb","Type":"ContainerDied","Data":"663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689"} Nov 02 03:50:04.638281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:04.638221 4868 scope.go:122] "RemoveContainer" containerID="663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689" Nov 02 03:50:04.642635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:04.642585 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:50:04.642959 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:04.642935 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:50:04.643161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:04.643139 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:50:05.647150 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:05.647112 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" event={"ID":"a01c3224-7761-4203-858f-c7934ed114fb","Type":"ContainerStarted","Data":"ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330"} Nov 02 03:50:06.063433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:06.059730 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:50:06.063433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:06.059859 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 03:50:06.063433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:06.059900 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 03:50:07.044808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:07.044768 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 03:50:07.045559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:07.045524 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 03:50:07.058632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:07.058595 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr"] Nov 02 03:50:07.098317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:07.098284 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68043d42-qq7tr"] Nov 02 03:50:08.045377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:08.042717 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 03:50:08.045377 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:08.042779 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 03:50:08.115371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:08.115319 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e33c5f-0128-4894-8995-1049eccd6e4e" path="/var/lib/kubelet/pods/d4e33c5f-0128-4894-8995-1049eccd6e4e/volumes" Nov 02 03:50:09.054198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:09.053714 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 03:50:09.054198 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:09.053797 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 03:50:10.053800 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:10.053761 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 03:50:10.054096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:10.054065 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 03:50:11.052376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:11.042454 4868 patch_prober.go:29] interesting pod/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 03:50:11.052376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:11.042528 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 03:50:12.046464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:12.046430 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:50:12.046866 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:12.046848 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:50:12.047456 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:12.047439 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 03:50:18.043974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:18.043943 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8"] Nov 02 03:50:18.094674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:18.094643 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab01c625-7wrl8"] Nov 02 03:50:18.116503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:18.116473 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb5e2a6-a7fc-468f-a9d9-925b7976d31e" path="/var/lib/kubelet/pods/1eb5e2a6-a7fc-468f-a9d9-925b7976d31e/volumes" Nov 02 03:50:19.043904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:19.043868 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx"] Nov 02 03:50:19.096491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:19.096456 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade77231-kjbbx"] Nov 02 03:50:20.115646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:20.115604 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17049507-8137-4f52-a24a-2ce6fe0715a2" path="/var/lib/kubelet/pods/17049507-8137-4f52-a24a-2ce6fe0715a2/volumes" Nov 02 03:50:25.047464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:25.047428 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h"] Nov 02 03:50:25.096761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:25.096732 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8bcaa7a-v6d5h"] Nov 02 03:50:26.116248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:26.116212 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d045d06c-4439-4f74-9f4c-e0904736433f" path="/var/lib/kubelet/pods/d045d06c-4439-4f74-9f4c-e0904736433f/volumes" Nov 02 03:50:33.446295 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:50:33.446261 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:50:39.321604 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:39.321573 4868 scope.go:122] "RemoveContainer" containerID="798ea1e204b199f47aa2231dba9ec26291a6bb690678e4d453ce0070428effc8" Nov 02 03:50:39.331318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:39.331300 4868 scope.go:122] "RemoveContainer" containerID="f9a125ead5344a377ff87cd690bc4d1c5c608b732563bba6d87bc9604c1ba696" Nov 02 03:50:39.339629 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:39.339610 4868 scope.go:122] "RemoveContainer" containerID="191d93bc14500b193f8140429547df3dd0899da4e6a3ec08a40632fa0bb8a7e0" Nov 02 03:50:39.348040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:39.348019 4868 scope.go:122] "RemoveContainer" containerID="e2e04f14acf8905d38e1cdd948ab8d209c3549136db6a5973e4ecd235a811551" Nov 02 03:50:39.358117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:39.358098 4868 scope.go:122] "RemoveContainer" containerID="0c5555a23ca3ec4b6ba1eb9403bf5de44219a93ce2b555753e3af82985b3cb7e" Nov 02 03:50:39.366076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:50:39.366041 4868 scope.go:122] "RemoveContainer" containerID="ea76ba97da38616d7f6b3ba7ae67da7eb71e9192e3626b3966cbe29a0bb4cd5f" Nov 02 03:51:03.446387 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:51:03.446337 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:51:07.511870 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:07.511832 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk"] Nov 02 03:51:07.512406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:07.512080 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" containerID="cri-o://95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7" gracePeriod=30 Nov 02 03:51:07.638683 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:07.638643 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nm8gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:51:07.638893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:07.638705 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:51:12.767343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.767311 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:51:12.841262 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841134 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841262 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841227 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841285 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841309 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841337 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841432 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841449 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841468 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841499 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841608 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:12.841918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841624 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:12.841918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841653 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data\") pod \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\" (UID: \"6e8dfdf1-69f9-418d-9455-75533b89a6ea\") " Nov 02 03:51:12.841918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841768 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:51:12.841918 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.841863 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:12.842191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.842114 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.842191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.842129 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.842191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.842138 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.842191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.842146 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.842191 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.842156 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.843833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.843798 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "kube-api-access-x7czw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:51:12.843833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.843806 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:51:12.843833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.843830 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:51:12.844010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.843914 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy" pod "6e8dfdf1-69f9-418d-9455-75533b89a6ea" (UID: "6e8dfdf1-69f9-418d-9455-75533b89a6ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:12.943769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.943734 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.943769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.943770 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.944013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.943785 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e8dfdf1-69f9-418d-9455-75533b89a6ea-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:12.944013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:12.943798 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7czw\" (UniqueName: \"kubernetes.io/projected/6e8dfdf1-69f9-418d-9455-75533b89a6ea-kube-api-access-x7czw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:13.139962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.139922 4868 generic.go:356] "Generic (PLEG): container finished" podID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerID="95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7" exitCode=0 Nov 02 03:51:13.140153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.139974 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" Nov 02 03:51:13.140153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.140014 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" event={"ID":"6e8dfdf1-69f9-418d-9455-75533b89a6ea","Type":"ContainerDied","Data":"95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7"} Nov 02 03:51:13.140153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.140074 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk" event={"ID":"6e8dfdf1-69f9-418d-9455-75533b89a6ea","Type":"ContainerDied","Data":"ab8ecf7b5482614cd38ef37fd283637634dcf14f7efbad3c70fc15351b4231cb"} Nov 02 03:51:13.140153 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.140106 4868 scope.go:122] "RemoveContainer" containerID="95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7" Nov 02 03:51:13.150902 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.150883 4868 scope.go:122] "RemoveContainer" containerID="95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7" Nov 02 03:51:13.151146 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:51:13.151129 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7\": container with ID starting with 95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7 not found: ID does not exist" containerID="95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7" Nov 02 03:51:13.151192 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.151153 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7"} err="failed to get container status \"95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7\": rpc error: code = NotFound desc = could not find container \"95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7\": container with ID starting with 95ed17b98207414cdb47525120d60ea9502e9161cd90849a040cb04f3919b4a7 not found: ID does not exist" Nov 02 03:51:13.302698 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.302662 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk"] Nov 02 03:51:13.306334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:13.306308 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nm8gk"] Nov 02 03:51:14.116882 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:14.116845 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" path="/var/lib/kubelet/pods/6e8dfdf1-69f9-418d-9455-75533b89a6ea/volumes" Nov 02 03:51:19.944126 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:19.944016 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p"] Nov 02 03:51:19.944613 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:19.944267 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" containerID="cri-o://ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902" gracePeriod=30 Nov 02 03:51:22.043731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.043694 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5"] Nov 02 03:51:22.044223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.044210 4868 cpu_manager.go:414] "containerMap: removing container" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" Nov 02 03:51:22.044281 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.044224 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" Nov 02 03:51:22.044322 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.044317 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc327185-945b-4338-ae87-1c38841a752f" containerName="istio-proxy" Nov 02 03:51:22.044373 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.044329 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8dfdf1-69f9-418d-9455-75533b89a6ea" containerName="istio-proxy" Nov 02 03:51:22.047892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.047872 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.103678 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.103641 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5"] Nov 02 03:51:22.145618 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.145577 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8p88\" (UniqueName: \"kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.145789 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.145769 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.145844 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.145813 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.145894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.145879 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.145954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.145903 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.146048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.146028 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.149265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.149244 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a7cf93\"/\"openshift-service-ca.crt\"" Nov 02 03:51:22.149265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.149244 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a7cf93\"/\"auth-test-model-a7cf93-kserve-self-signed-certs\"" Nov 02 03:51:22.149492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.149301 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a7cf93\"/\"default-dockercfg-w5sdv\"" Nov 02 03:51:22.149492 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.149408 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a7cf93\"/\"kube-root-ca.crt\"" Nov 02 03:51:22.236227 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.236187 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts"] Nov 02 03:51:22.236476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.236443 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" containerID="cri-o://b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c" gracePeriod=30 Nov 02 03:51:22.247483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.247448 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.247624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.247491 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.247624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.247583 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.247743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.247635 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8p88\" (UniqueName: \"kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.247801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.247792 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.247861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.247843 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.248286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.248260 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.248449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.248316 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.249020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.248994 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.251513 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.251490 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.252483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.252464 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.309828 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.309758 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8p88\" (UniqueName: \"kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88\") pod \"auth-test-model-a7cf93-kserve-59687b989c-sffm5\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.360616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.360585 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:22.749447 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:22.749421 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5"] Nov 02 03:51:22.751109 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:51:22.751078 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aff6c10_c21e_4453_ad0d_8abcdd626812.slice/crio-b018307e632225d9183fc9d5a8dfd439a169542bbd66d5cd95919b51f35d1dfb WatchSource:0}: Error finding container b018307e632225d9183fc9d5a8dfd439a169542bbd66d5cd95919b51f35d1dfb: Status 404 returned error can't find the container with id b018307e632225d9183fc9d5a8dfd439a169542bbd66d5cd95919b51f35d1dfb Nov 02 03:51:23.246997 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:23.246959 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" event={"ID":"7aff6c10-c21e-4453-ad0d-8abcdd626812","Type":"ContainerStarted","Data":"261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3"} Nov 02 03:51:23.247426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:23.247008 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" event={"ID":"7aff6c10-c21e-4453-ad0d-8abcdd626812","Type":"ContainerStarted","Data":"b018307e632225d9183fc9d5a8dfd439a169542bbd66d5cd95919b51f35d1dfb"} Nov 02 03:51:23.361533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:23.361494 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:23.361533 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:23.361478 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" podStartSLOduration=1.361463875 podStartE2EDuration="1.361463875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:51:23.360024799 +0000 UTC m=+3825.355725743" watchObservedRunningTime="2026-11-02 03:51:23.361463875 +0000 UTC m=+3825.357164817" Nov 02 03:51:25.221141 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.221075 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:51:25.260029 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.259999 4868 generic.go:356] "Generic (PLEG): container finished" podID="40bbd771-6e3c-4747-97de-8ae538409968" containerID="ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902" exitCode=0 Nov 02 03:51:25.260167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.260048 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" Nov 02 03:51:25.260167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.260075 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" event={"ID":"40bbd771-6e3c-4747-97de-8ae538409968","Type":"ContainerDied","Data":"ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902"} Nov 02 03:51:25.260167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.260130 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p" event={"ID":"40bbd771-6e3c-4747-97de-8ae538409968","Type":"ContainerDied","Data":"c6e77b9f4a6c24f4432db192f004d7a6a24e07f15464835366cbcb8d631295b3"} Nov 02 03:51:25.260167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.260157 4868 scope.go:122] "RemoveContainer" containerID="ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902" Nov 02 03:51:25.270139 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.270115 4868 scope.go:122] "RemoveContainer" containerID="ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902" Nov 02 03:51:25.270424 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:51:25.270397 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902\": container with ID starting with ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902 not found: ID does not exist" containerID="ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902" Nov 02 03:51:25.270488 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.270434 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902"} err="failed to get container status \"ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902\": rpc error: code = NotFound desc = could not find container \"ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902\": container with ID starting with ead7cf2c22effa033396dcaf0c1fc73c8422fdadf0f46d7a167c707ffabdf902 not found: ID does not exist" Nov 02 03:51:25.396631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396602 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.396803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396644 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.396803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396772 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.396942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396874 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.396942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396927 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.397053 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396959 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:25.397053 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396969 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.397053 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.396989 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:51:25.397053 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397007 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.397261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397087 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.397261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397166 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48\") pod \"40bbd771-6e3c-4747-97de-8ae538409968\" (UID: \"40bbd771-6e3c-4747-97de-8ae538409968\") " Nov 02 03:51:25.397261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397250 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:25.397434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397281 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:25.397434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397333 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:25.397784 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397760 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.397892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397787 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.397892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397803 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.397892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397817 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bbd771-6e3c-4747-97de-8ae538409968-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.397892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.397830 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.399470 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.399449 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:51:25.399852 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.399820 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:25.399934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.399888 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:51:25.401612 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.401590 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48" pod "40bbd771-6e3c-4747-97de-8ae538409968" (UID: "40bbd771-6e3c-4747-97de-8ae538409968"). InnerVolumeSpecName "kube-api-access-sjq48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:51:25.498808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.498740 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bbd771-6e3c-4747-97de-8ae538409968-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.498808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.498767 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.498808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.498777 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjq48\" (UniqueName: \"kubernetes.io/projected/40bbd771-6e3c-4747-97de-8ae538409968-kube-api-access-sjq48\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.498808 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.498787 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bbd771-6e3c-4747-97de-8ae538409968-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:25.680958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.680922 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p"] Nov 02 03:51:25.683959 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.683928 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-8tv5p"] Nov 02 03:51:25.772499 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.772417 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz"] Nov 02 03:51:25.773189 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.773170 4868 cpu_manager.go:414] "containerMap: removing container" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" Nov 02 03:51:25.773235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.773194 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" Nov 02 03:51:25.773383 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.773343 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bbd771-6e3c-4747-97de-8ae538409968" containerName="istio-proxy" Nov 02 03:51:25.778304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.778282 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:25.836025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.835985 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz"] Nov 02 03:51:25.836221 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.836204 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-7c1bee\"/\"default-dockercfg-q8tql\"" Nov 02 03:51:25.836913 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.836893 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-7c1bee\"/\"auth-test-model-7c1bee-kserve-self-signed-certs\"" Nov 02 03:51:25.837022 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.837008 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-7c1bee\"/\"kube-root-ca.crt\"" Nov 02 03:51:25.837111 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.837098 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-7c1bee\"/\"openshift-service-ca.crt\"" Nov 02 03:51:25.902775 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.902743 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:25.902963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.902795 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:25.902963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.902861 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psbjg\" (UniqueName: \"kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:25.902963 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.902938 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:25.903128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.903004 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:25.903128 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:25.903026 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.004687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.004653 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.004687 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.004691 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.004972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.004764 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.004972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.004817 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.004972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.004847 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psbjg\" (UniqueName: \"kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.004972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.004894 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.005223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.005149 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.005223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.005192 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.005329 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.005295 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.007286 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.007264 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.007621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.007602 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.065706 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.065631 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psbjg\" (UniqueName: \"kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg\") pod \"auth-test-model-7c1bee-kserve-9854985c7-xt9fz\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.092364 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.092327 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:26.118002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.117974 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bbd771-6e3c-4747-97de-8ae538409968" path="/var/lib/kubelet/pods/40bbd771-6e3c-4747-97de-8ae538409968/volumes" Nov 02 03:51:26.485574 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:26.485544 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz"] Nov 02 03:51:26.486762 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:51:26.486722 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c94f5ac_d6b1_4e4a_ac96_5e88c6530898.slice/crio-c6d617fc27fde381ece623f17478ad79204eee5aa2c2cb4a286870a43d291341 WatchSource:0}: Error finding container c6d617fc27fde381ece623f17478ad79204eee5aa2c2cb4a286870a43d291341: Status 404 returned error can't find the container with id c6d617fc27fde381ece623f17478ad79204eee5aa2c2cb4a286870a43d291341 Nov 02 03:51:27.278771 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.278735 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" event={"ID":"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898","Type":"ContainerStarted","Data":"9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7"} Nov 02 03:51:27.278889 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.278799 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" event={"ID":"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898","Type":"ContainerStarted","Data":"c6d617fc27fde381ece623f17478ad79204eee5aa2c2cb4a286870a43d291341"} Nov 02 03:51:27.393647 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.393583 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" podStartSLOduration=2.393563488 podStartE2EDuration="2.393563488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:51:27.392852061 +0000 UTC m=+3829.388553020" watchObservedRunningTime="2026-11-02 03:51:27.393563488 +0000 UTC m=+3829.389264432" Nov 02 03:51:27.530116 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.530036 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:51:27.537306 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537281 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537318 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537372 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537450 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537434 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537628 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537476 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537628 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537519 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537749 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537645 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537749 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537681 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.537861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537833 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:51:27.537861 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.537828 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:27.538103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.538079 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:27.538187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.538136 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:27.538240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.538211 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.538240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.538226 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3909121-92f8-4580-a841-2756399cfed5-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.538240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.538237 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.539892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.539864 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:27.540037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.539974 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "kube-api-access-7qj9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:51:27.540037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.540021 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:51:27.540154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.540063 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:51:27.639023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.638991 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket\") pod \"c3909121-92f8-4580-a841-2756399cfed5\" (UID: \"c3909121-92f8-4580-a841-2756399cfed5\") " Nov 02 03:51:27.639249 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639229 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket" pod "c3909121-92f8-4580-a841-2756399cfed5" (UID: "c3909121-92f8-4580-a841-2756399cfed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:51:27.639500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639485 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.639541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639505 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.639541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639521 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3909121-92f8-4580-a841-2756399cfed5-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.639541 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639536 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.639656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639551 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qj9t\" (UniqueName: \"kubernetes.io/projected/c3909121-92f8-4580-a841-2756399cfed5-kube-api-access-7qj9t\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:27.639656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:27.639565 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3909121-92f8-4580-a841-2756399cfed5-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:51:28.092634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.092601 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:28.286676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.286637 4868 generic.go:356] "Generic (PLEG): container finished" podID="c3909121-92f8-4580-a841-2756399cfed5" containerID="b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c" exitCode=0 Nov 02 03:51:28.286841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.286696 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" Nov 02 03:51:28.286841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.286723 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" event={"ID":"c3909121-92f8-4580-a841-2756399cfed5","Type":"ContainerDied","Data":"b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c"} Nov 02 03:51:28.286841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.286775 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts" event={"ID":"c3909121-92f8-4580-a841-2756399cfed5","Type":"ContainerDied","Data":"0754c181de8ab117296de44bf0251b7da8104d979531c428fe4ebe605a2d3f6b"} Nov 02 03:51:28.286841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.286798 4868 scope.go:122] "RemoveContainer" containerID="b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c" Nov 02 03:51:28.296661 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.296635 4868 scope.go:122] "RemoveContainer" containerID="b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c" Nov 02 03:51:28.296975 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:51:28.296939 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c\": container with ID starting with b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c not found: ID does not exist" containerID="b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c" Nov 02 03:51:28.297093 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.296982 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c"} err="failed to get container status \"b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c\": rpc error: code = NotFound desc = could not find container \"b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c\": container with ID starting with b042c4335c61cba40ed3e85f03acf0dc89ce307cb9023e9b1423ba2b9f5c149c not found: ID does not exist" Nov 02 03:51:28.448731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.448695 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts"] Nov 02 03:51:28.498048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.498019 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f6e39-openshift-default-55b56b9965-hspts"] Nov 02 03:51:28.815432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.814476 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:51:28.815432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:28.814575 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:51:30.115304 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:30.115273 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3909121-92f8-4580-a841-2756399cfed5" path="/var/lib/kubelet/pods/c3909121-92f8-4580-a841-2756399cfed5/volumes" Nov 02 03:51:32.360738 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:32.360704 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:32.373017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:32.372989 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:33.330082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:33.330052 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:51:33.447433 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:51:33.447393 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:51:36.092744 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:36.092714 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:36.105166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:36.105139 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:36.348884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:36.348808 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:51:39.412306 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.412271 4868 scope.go:122] "RemoveContainer" containerID="69631582e6bd28ff946def39f1efecda2f7496cc0f04ebed014584a9101f9245" Nov 02 03:51:39.421988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.421965 4868 scope.go:122] "RemoveContainer" containerID="4a55e18a89aabd15d9f77922b99dc853a3c7d1634a1b58c8f76f23a52dbe0207" Nov 02 03:51:39.431046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.431029 4868 scope.go:122] "RemoveContainer" containerID="46d33280db72e44c13156a33287dff50f499bd53f480053cfb1c2787c5704720" Nov 02 03:51:39.439155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.439129 4868 scope.go:122] "RemoveContainer" containerID="4c4f77ac87e576c55f6c793b249c97661ee7a664ab2a3af00bd3abbc4afc52d3" Nov 02 03:51:39.447384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.447341 4868 scope.go:122] "RemoveContainer" containerID="760cbadfedc17ef6fcb03c08a6efd095d4bfe7262363d6c5f83a6c876be2175b" Nov 02 03:51:39.455519 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.455502 4868 scope.go:122] "RemoveContainer" containerID="da6eee9bf2fcacd235e6dbe5ad8ea612ca3d82ddc175e25c3cb7216b83b4fd7c" Nov 02 03:51:39.463260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:39.463243 4868 scope.go:122] "RemoveContainer" containerID="01cea48d88bdb2cbadc9425b57e6541d5c68b5c05cc97b1aa664c39abb09add4" Nov 02 03:51:46.461752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.461707 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6"] Nov 02 03:51:46.462225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.462205 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" Nov 02 03:51:46.462225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.462216 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" Nov 02 03:51:46.462330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.462318 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3909121-92f8-4580-a841-2756399cfed5" containerName="istio-proxy" Nov 02 03:51:46.465763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.465742 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.567920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.567890 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aad578\"/\"kube-root-ca.crt\"" Nov 02 03:51:46.568133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.568022 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aad578\"/\"auth-test-model-aad578-kserve-self-signed-certs\"" Nov 02 03:51:46.568133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.568056 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aad578\"/\"openshift-service-ca.crt\"" Nov 02 03:51:46.568133 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.568124 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aad578\"/\"default-dockercfg-lqcpf\"" Nov 02 03:51:46.570776 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.570752 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6"] Nov 02 03:51:46.585507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.585483 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2gh\" (UniqueName: \"kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.585630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.585533 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.585706 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.585659 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.585778 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.585733 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.585827 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.585818 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.585868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.585859 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.689374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.688943 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.689374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.689004 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.689374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.689050 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.689374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.689113 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.689374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.689160 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.689374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.689295 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2gh\" (UniqueName: \"kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.691100 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.691069 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.691660 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.691635 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.692109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.692061 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.702367 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.699509 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.703048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.703003 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.754154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.754077 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2gh\" (UniqueName: \"kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh\") pod \"auth-test-model-aad578-kserve-bf555c7d9-kr8b6\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:46.778392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:46.778341 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:47.160222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:47.160192 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6"] Nov 02 03:51:47.162463 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:51:47.162433 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd79d205c_ad61_4efe_a221_cb59738dedb0.slice/crio-6046b27cf60b255ccb131fca3bab30ef1442f254ac67577c12db323149efea70 WatchSource:0}: Error finding container 6046b27cf60b255ccb131fca3bab30ef1442f254ac67577c12db323149efea70: Status 404 returned error can't find the container with id 6046b27cf60b255ccb131fca3bab30ef1442f254ac67577c12db323149efea70 Nov 02 03:51:47.405955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:47.405848 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" event={"ID":"d79d205c-ad61-4efe-a221-cb59738dedb0","Type":"ContainerStarted","Data":"5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30"} Nov 02 03:51:47.405955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:47.405907 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" event={"ID":"d79d205c-ad61-4efe-a221-cb59738dedb0","Type":"ContainerStarted","Data":"6046b27cf60b255ccb131fca3bab30ef1442f254ac67577c12db323149efea70"} Nov 02 03:51:47.520099 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:47.520048 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" podStartSLOduration=1.520033057 podStartE2EDuration="1.520033057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:51:47.51876996 +0000 UTC m=+3849.514470902" watchObservedRunningTime="2026-11-02 03:51:47.520033057 +0000 UTC m=+3849.515734039" Nov 02 03:51:47.779136 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:47.779056 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:51.432448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:51.432415 4868 generic.go:356] "Generic (PLEG): container finished" podID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerID="0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc" exitCode=0 Nov 02 03:51:51.432908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:51.432488 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" event={"ID":"7d5c9167-8e93-4fdc-871d-2d160226364d","Type":"ContainerDied","Data":"0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc"} Nov 02 03:51:51.432955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:51.432908 4868 scope.go:122] "RemoveContainer" containerID="0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc" Nov 02 03:51:51.434452 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:51.434423 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:51:51.434575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:51.434502 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:51:51.434575 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:51.434555 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:51:52.462270 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:52.462235 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" event={"ID":"7d5c9167-8e93-4fdc-871d-2d160226364d","Type":"ContainerStarted","Data":"968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151"} Nov 02 03:51:53.029052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:53.029018 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:51:53.029269 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:53.029249 4868 patch_prober.go:29] interesting pod/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 03:51:53.029319 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:53.029293 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 03:51:54.029338 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:54.029303 4868 patch_prober.go:29] interesting pod/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 03:51:54.029760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:54.029380 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 03:51:55.029232 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:55.029197 4868 patch_prober.go:29] interesting pod/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 03:51:55.029454 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:55.029255 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 03:51:56.029892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:56.029854 4868 patch_prober.go:29] interesting pod/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 03:51:56.030297 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:56.029910 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 03:51:56.779068 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:56.778959 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:56.791832 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:56.791794 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:57.029542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:57.029462 4868 patch_prober.go:29] interesting pod/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 03:51:57.029542 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:57.029520 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 03:51:57.502600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:57.502567 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:51:58.034942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:58.034908 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:51:58.035317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:58.035301 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:51:58.035981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:51:58.035964 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:52:03.445977 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:52:03.445937 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:52:15.045718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:52:15.045682 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5"] Nov 02 03:52:15.095380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:52:15.095337 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f885532-22qn5"] Nov 02 03:52:16.140223 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:52:16.140188 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3eec97-6aa4-482a-b10f-62b499481b38" path="/var/lib/kubelet/pods/8e3eec97-6aa4-482a-b10f-62b499481b38/volumes" Nov 02 03:52:33.445671 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:52:33.445637 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:52:38.611521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:52:38.611491 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:52:38.619672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:52:38.619650 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:52:39.525642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:52:39.525616 4868 scope.go:122] "RemoveContainer" containerID="d521333e1617488c4bf4cc0f154433dba82f7b2ea11f5c5170b021b657228d31" Nov 02 03:53:03.446979 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:53:03.446940 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:53:33.446231 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:53:33.446194 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:53:39.549122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:53:39.549089 4868 scope.go:122] "RemoveContainer" containerID="59ab22d53890ca079f08b5e815ac23a7ae97efd1d2ec3704048ab3656d5825b2" Nov 02 03:54:03.446146 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:54:03.446110 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:54:09.261767 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.261732 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6"] Nov 02 03:54:09.265679 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.265660 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.367613 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.367579 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-86f4c4\"/\"default-dockercfg-2f67t\"" Nov 02 03:54:09.367851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.367830 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-86f4c4\"/\"kube-root-ca.crt\"" Nov 02 03:54:09.367851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.367831 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-86f4c4\"/\"openshift-service-ca.crt\"" Nov 02 03:54:09.367973 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.367864 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-86f4c4\"/\"explicit-model-86f4c4-kserve-self-signed-certs\"" Nov 02 03:54:09.370386 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.370339 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6"] Nov 02 03:54:09.403530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.403447 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.403923 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.403824 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.404082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.403904 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lcn\" (UniqueName: \"kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.404314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.404257 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.404529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.404474 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.404768 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.404665 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506004 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506089 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506118 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lcn\" (UniqueName: \"kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506167 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506199 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506236 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506477 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506547 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.506747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.506728 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.508886 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.508863 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.509160 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.509139 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.562311 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.562241 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lcn\" (UniqueName: \"kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn\") pod \"explicit-model-86f4c4-kserve-84fd78dd69-pg4n6\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.577230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.577203 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:09.954156 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:09.954128 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6"] Nov 02 03:54:09.955636 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:54:09.955604 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a47a048_3cff_47cd_b470_cb574ea0670b.slice/crio-36038bf7ae3d91638fdc6fdbcb146b5a04499be546cdb1480b84d213adb6a29d WatchSource:0}: Error finding container 36038bf7ae3d91638fdc6fdbcb146b5a04499be546cdb1480b84d213adb6a29d: Status 404 returned error can't find the container with id 36038bf7ae3d91638fdc6fdbcb146b5a04499be546cdb1480b84d213adb6a29d Nov 02 03:54:10.303745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:10.303657 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" event={"ID":"3a47a048-3cff-47cd-b470-cb574ea0670b","Type":"ContainerStarted","Data":"883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f"} Nov 02 03:54:10.303745 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:10.303704 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" event={"ID":"3a47a048-3cff-47cd-b470-cb574ea0670b","Type":"ContainerStarted","Data":"36038bf7ae3d91638fdc6fdbcb146b5a04499be546cdb1480b84d213adb6a29d"} Nov 02 03:54:10.417002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:10.416953 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" podStartSLOduration=1.416940696 podStartE2EDuration="1.416940696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:54:10.415667022 +0000 UTC m=+3992.411367980" watchObservedRunningTime="2026-11-02 03:54:10.416940696 +0000 UTC m=+3992.412641661" Nov 02 03:54:10.577854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:10.577772 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:19.585280 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:19.585181 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:19.603503 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:19.603477 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:20.406384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:20.406339 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:25.832166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:25.832127 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6"] Nov 02 03:54:26.318454 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.318420 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb"] Nov 02 03:54:26.322384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.322366 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.361910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.361885 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tnf\" (UniqueName: \"kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.362054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.361931 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.362054 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.362022 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.375200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.375180 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:54:26.375415 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.375376 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:54:26.378454 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.378432 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb"] Nov 02 03:54:26.462762 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.462729 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tnf\" (UniqueName: \"kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.462954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.462775 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.462954 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.462806 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.463608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.463582 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.465305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.465282 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.518677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.518652 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tnf\" (UniqueName: \"kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf\") pod \"maas-api-revoke-keys-e2e-explicit-86f4c4-86drb\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:26.637908 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:26.637877 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:27.012151 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:27.012128 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb"] Nov 02 03:54:27.014544 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:54:27.014518 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf6c38b_c496_45e8_bd2b_20a83cc67f36.slice/crio-3b66df6a9ec4728f8d38e6c559d14c4dd2efcdc8168f8e69189f78ad319fee3d WatchSource:0}: Error finding container 3b66df6a9ec4728f8d38e6c559d14c4dd2efcdc8168f8e69189f78ad319fee3d: Status 404 returned error can't find the container with id 3b66df6a9ec4728f8d38e6c559d14c4dd2efcdc8168f8e69189f78ad319fee3d Nov 02 03:54:27.437900 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:27.437864 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" event={"ID":"dcf6c38b-c496-45e8-bd2b-20a83cc67f36","Type":"ContainerStarted","Data":"b42a92de74b9d59588dde2bdb40fb76020818b74b17b1e881f8441345d43882e"} Nov 02 03:54:27.438075 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:27.437910 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" event={"ID":"dcf6c38b-c496-45e8-bd2b-20a83cc67f36","Type":"ContainerStarted","Data":"3b66df6a9ec4728f8d38e6c559d14c4dd2efcdc8168f8e69189f78ad319fee3d"} Nov 02 03:54:27.550654 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:27.550603 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" podStartSLOduration=1.550587683 podStartE2EDuration="1.550587683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:54:27.54937458 +0000 UTC m=+4009.545075523" watchObservedRunningTime="2026-11-02 03:54:27.550587683 +0000 UTC m=+4009.546288626" Nov 02 03:54:28.445989 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:28.445955 4868 generic.go:356] "Generic (PLEG): container finished" podID="dcf6c38b-c496-45e8-bd2b-20a83cc67f36" containerID="b42a92de74b9d59588dde2bdb40fb76020818b74b17b1e881f8441345d43882e" exitCode=0 Nov 02 03:54:28.446438 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:28.446037 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" event={"ID":"dcf6c38b-c496-45e8-bd2b-20a83cc67f36","Type":"ContainerDied","Data":"b42a92de74b9d59588dde2bdb40fb76020818b74b17b1e881f8441345d43882e"} Nov 02 03:54:29.603405 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.603378 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:29.695207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.695176 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf\" (UniqueName: \"kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf\") pod \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " Nov 02 03:54:29.695379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.695212 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca\") pod \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " Nov 02 03:54:29.695379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.695313 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca\") pod \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\" (UID: \"dcf6c38b-c496-45e8-bd2b-20a83cc67f36\") " Nov 02 03:54:29.695613 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.695592 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca" pod "dcf6c38b-c496-45e8-bd2b-20a83cc67f36" (UID: "dcf6c38b-c496-45e8-bd2b-20a83cc67f36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:54:29.695723 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.695709 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:29.697402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.697380 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca" pod "dcf6c38b-c496-45e8-bd2b-20a83cc67f36" (UID: "dcf6c38b-c496-45e8-bd2b-20a83cc67f36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:54:29.697460 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.697396 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf" pod "dcf6c38b-c496-45e8-bd2b-20a83cc67f36" (UID: "dcf6c38b-c496-45e8-bd2b-20a83cc67f36"). InnerVolumeSpecName "kube-api-access-x8tnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:54:29.796915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.796882 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:29.796915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:29.796912 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8tnf\" (UniqueName: \"kubernetes.io/projected/dcf6c38b-c496-45e8-bd2b-20a83cc67f36-kube-api-access-x8tnf\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:30.459479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:30.459394 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" event={"ID":"dcf6c38b-c496-45e8-bd2b-20a83cc67f36","Type":"ContainerDied","Data":"3b66df6a9ec4728f8d38e6c559d14c4dd2efcdc8168f8e69189f78ad319fee3d"} Nov 02 03:54:30.459479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:30.459414 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb" Nov 02 03:54:30.459479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:30.459452 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b66df6a9ec4728f8d38e6c559d14c4dd2efcdc8168f8e69189f78ad319fee3d" Nov 02 03:54:33.446126 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:54:33.446087 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:54:40.853013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:40.852968 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" podUID="3a47a048-3cff-47cd-b470-cb574ea0670b" containerName="main" containerID="cri-o://883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f" gracePeriod=45 Nov 02 03:54:41.131484 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.131461 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:41.198462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198437 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm\") pod \"3a47a048-3cff-47cd-b470-cb574ea0670b\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " Nov 02 03:54:41.198627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198479 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs\") pod \"3a47a048-3cff-47cd-b470-cb574ea0670b\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " Nov 02 03:54:41.198627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198588 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home\") pod \"3a47a048-3cff-47cd-b470-cb574ea0670b\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " Nov 02 03:54:41.198766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198656 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache\") pod \"3a47a048-3cff-47cd-b470-cb574ea0670b\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " Nov 02 03:54:41.198766 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198686 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir\") pod \"3a47a048-3cff-47cd-b470-cb574ea0670b\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " Nov 02 03:54:41.198876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198797 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn\" (UniqueName: \"kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn\") pod \"3a47a048-3cff-47cd-b470-cb574ea0670b\" (UID: \"3a47a048-3cff-47cd-b470-cb574ea0670b\") " Nov 02 03:54:41.198876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198848 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home" pod "3a47a048-3cff-47cd-b470-cb574ea0670b" (UID: "3a47a048-3cff-47cd-b470-cb574ea0670b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:54:41.198999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198903 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache" pod "3a47a048-3cff-47cd-b470-cb574ea0670b" (UID: "3a47a048-3cff-47cd-b470-cb574ea0670b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:54:41.198999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.198980 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir" pod "3a47a048-3cff-47cd-b470-cb574ea0670b" (UID: "3a47a048-3cff-47cd-b470-cb574ea0670b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:54:41.199310 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.199289 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:41.199451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.199317 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:41.199451 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.199334 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:41.200846 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.200817 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs" pod "3a47a048-3cff-47cd-b470-cb574ea0670b" (UID: "3a47a048-3cff-47cd-b470-cb574ea0670b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:54:41.200927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.200883 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn" pod "3a47a048-3cff-47cd-b470-cb574ea0670b" (UID: "3a47a048-3cff-47cd-b470-cb574ea0670b"). InnerVolumeSpecName "kube-api-access-m2lcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:54:41.200927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.200906 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm" pod "3a47a048-3cff-47cd-b470-cb574ea0670b" (UID: "3a47a048-3cff-47cd-b470-cb574ea0670b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:54:41.300782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.300733 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lcn\" (UniqueName: \"kubernetes.io/projected/3a47a048-3cff-47cd-b470-cb574ea0670b-kube-api-access-m2lcn\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:41.300782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.300763 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a47a048-3cff-47cd-b470-cb574ea0670b-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:41.300782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.300773 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a47a048-3cff-47cd-b470-cb574ea0670b-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:54:41.532735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.532648 4868 generic.go:356] "Generic (PLEG): container finished" podID="3a47a048-3cff-47cd-b470-cb574ea0670b" containerID="883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f" exitCode=0 Nov 02 03:54:41.532735 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.532689 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" Nov 02 03:54:41.532919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.532740 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" event={"ID":"3a47a048-3cff-47cd-b470-cb574ea0670b","Type":"ContainerDied","Data":"883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f"} Nov 02 03:54:41.532919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.532780 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6" event={"ID":"3a47a048-3cff-47cd-b470-cb574ea0670b","Type":"ContainerDied","Data":"36038bf7ae3d91638fdc6fdbcb146b5a04499be546cdb1480b84d213adb6a29d"} Nov 02 03:54:41.532919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.532800 4868 scope.go:122] "RemoveContainer" containerID="883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f" Nov 02 03:54:41.543242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.543225 4868 scope.go:122] "RemoveContainer" containerID="883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f" Nov 02 03:54:41.543538 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:54:41.543519 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f\": container with ID starting with 883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f not found: ID does not exist" containerID="883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f" Nov 02 03:54:41.543597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.543546 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f"} err="failed to get container status \"883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f\": rpc error: code = NotFound desc = could not find container \"883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f\": container with ID starting with 883cfe43ba1d6acc283216abc6d17e286327a4935ebdbbbd796b46431b08cd0f not found: ID does not exist" Nov 02 03:54:41.696331 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.696300 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6"] Nov 02 03:54:41.745865 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:41.745833 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-86f4c4/explicit-model-86f4c4-kserve-84fd78dd69-pg4n6"] Nov 02 03:54:42.115036 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:54:42.115006 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a47a048-3cff-47cd-b470-cb574ea0670b" path="/var/lib/kubelet/pods/3a47a048-3cff-47cd-b470-cb574ea0670b/volumes" Nov 02 03:55:03.446215 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:55:03.446176 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:55:29.729038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:29.729003 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb"] Nov 02 03:55:29.778115 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:29.778086 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-86f4c4-86drb"] Nov 02 03:55:30.114474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:30.114441 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf6c38b-c496-45e8-bd2b-20a83cc67f36" path="/var/lib/kubelet/pods/dcf6c38b-c496-45e8-bd2b-20a83cc67f36/volumes" Nov 02 03:55:33.446803 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:55:33.446767 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:55:36.026225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:36.026172 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln"] Nov 02 03:55:36.026775 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:36.026437 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" containerID="cri-o://968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151" gracePeriod=30 Nov 02 03:55:36.908361 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:36.908319 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2"] Nov 02 03:55:36.908591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:36.908563 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" containerID="cri-o://3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1" gracePeriod=30 Nov 02 03:55:39.595589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:39.595556 4868 scope.go:122] "RemoveContainer" containerID="887faa2d3d50df6c24b2b430e075055397d0a3ca1b74a3ea28c39c53b331c574" Nov 02 03:55:39.965274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:39.965236 4868 patch_prober.go:29] interesting pod/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 03:55:39.965466 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:39.965297 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 03:55:41.300976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.300915 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:55:41.447699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447669 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.447898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447731 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.447898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447753 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.447898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447822 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.447898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447852 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.447898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447881 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.448172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447939 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.448172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.447971 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.448172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448070 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq\") pod \"7d5c9167-8e93-4fdc-871d-2d160226364d\" (UID: \"7d5c9167-8e93-4fdc-871d-2d160226364d\") " Nov 02 03:55:41.448172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448082 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:41.448172 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448139 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:41.448472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448244 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:41.448527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448462 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:41.448582 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448539 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:55:41.448734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448716 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.448806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448741 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.448806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448757 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5c9167-8e93-4fdc-871d-2d160226364d-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.448806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448770 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.448806 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.448785 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.450168 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.450144 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:55:41.450715 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.450688 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:41.452431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.450748 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:55:41.452431 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.450791 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq" pod "7d5c9167-8e93-4fdc-871d-2d160226364d" (UID: "7d5c9167-8e93-4fdc-871d-2d160226364d"). InnerVolumeSpecName "kube-api-access-884zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:55:41.550340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.550309 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.550340 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.550338 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.550572 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.550347 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5c9167-8e93-4fdc-871d-2d160226364d-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.550572 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.550381 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-884zq\" (UniqueName: \"kubernetes.io/projected/7d5c9167-8e93-4fdc-871d-2d160226364d-kube-api-access-884zq\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:41.895300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.895267 4868 generic.go:356] "Generic (PLEG): container finished" podID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerID="968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151" exitCode=0 Nov 02 03:55:41.895532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.895321 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" Nov 02 03:55:41.895532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.895359 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" event={"ID":"7d5c9167-8e93-4fdc-871d-2d160226364d","Type":"ContainerDied","Data":"968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151"} Nov 02 03:55:41.895532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.895444 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln" event={"ID":"7d5c9167-8e93-4fdc-871d-2d160226364d","Type":"ContainerDied","Data":"78aba9fa38d3ed41d6959e8e4cfb1f105cfca924f35dc762bcceea8844cf7b75"} Nov 02 03:55:41.895532 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.895466 4868 scope.go:122] "RemoveContainer" containerID="968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151" Nov 02 03:55:41.907145 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.907129 4868 scope.go:122] "RemoveContainer" containerID="0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc" Nov 02 03:55:41.918891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.918867 4868 scope.go:122] "RemoveContainer" containerID="968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151" Nov 02 03:55:41.919176 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:55:41.919158 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151\": container with ID starting with 968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151 not found: ID does not exist" containerID="968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151" Nov 02 03:55:41.919237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.919186 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151"} err="failed to get container status \"968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151\": rpc error: code = NotFound desc = could not find container \"968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151\": container with ID starting with 968a8b796d599cbff6e79f903e1c47b992593ee4a7159ca384c8fa7a23c9c151 not found: ID does not exist" Nov 02 03:55:41.919237 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.919205 4868 scope.go:122] "RemoveContainer" containerID="0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc" Nov 02 03:55:41.920500 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:55:41.920467 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc\": container with ID starting with 0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc not found: ID does not exist" containerID="0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc" Nov 02 03:55:41.920599 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:41.920506 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc"} err="failed to get container status \"0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc\": rpc error: code = NotFound desc = could not find container \"0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc\": container with ID starting with 0d6800e26d2757484a7aec4402df14dde8e6ca6a2126b9d021a16a5635bcaafc not found: ID does not exist" Nov 02 03:55:42.061033 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.061002 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln"] Nov 02 03:55:42.119170 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.119130 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-86f4c4-openshift-default-66f6f5bb45-68xln"] Nov 02 03:55:42.168112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.168089 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:55:42.257926 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.257892 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258094 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.257949 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258094 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258006 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258092 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258140 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258222 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258198 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258236 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258275 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258284 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:42.258399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258304 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket\") pod \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\" (UID: \"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2\") " Nov 02 03:55:42.258597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258449 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:42.258805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258783 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:42.258898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258789 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.258898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258831 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.259026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.258983 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:42.259085 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.259056 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:55:42.260676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.260651 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:55:42.260791 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.260753 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:55:42.260910 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.260892 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 03:55:42.261020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.261002 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h" pod "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" (UID: "9fa41d8c-9720-4f2e-8b2e-d5891138c9f2"). InnerVolumeSpecName "kube-api-access-pq95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:55:42.359442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359409 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.359442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359437 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.359442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359447 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.360049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359455 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.360049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359465 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.360049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359476 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.360049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.359490 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq95h\" (UniqueName: \"kubernetes.io/projected/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2-kube-api-access-pq95h\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:55:42.902343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.902310 4868 generic.go:356] "Generic (PLEG): container finished" podID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerID="3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1" exitCode=0 Nov 02 03:55:42.902559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.902382 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" event={"ID":"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2","Type":"ContainerDied","Data":"3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1"} Nov 02 03:55:42.902559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.902441 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" event={"ID":"9fa41d8c-9720-4f2e-8b2e-d5891138c9f2","Type":"ContainerDied","Data":"4dd445c08da92394bb82e38d6b6ab6c2cc9e98549b19611ed3d3510ca1982e80"} Nov 02 03:55:42.902559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.902387 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2" Nov 02 03:55:42.902559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.902475 4868 scope.go:122] "RemoveContainer" containerID="3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1" Nov 02 03:55:42.913773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.913757 4868 scope.go:122] "RemoveContainer" containerID="3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1" Nov 02 03:55:42.914060 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:55:42.914034 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1\": container with ID starting with 3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1 not found: ID does not exist" containerID="3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1" Nov 02 03:55:42.914114 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:42.914072 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1"} err="failed to get container status \"3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1\": rpc error: code = NotFound desc = could not find container \"3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1\": container with ID starting with 3ca031ca021b8df9b1678d4464a52bcfa606e01cc7ef0e14855351bf405559d1 not found: ID does not exist" Nov 02 03:55:43.067247 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.067212 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2"] Nov 02 03:55:43.116754 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.116723 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-858236-openshift-default-74fcd64bf5-twfd2"] Nov 02 03:55:43.568119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568087 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px"] Nov 02 03:55:43.568560 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568546 4868 cpu_manager.go:414] "containerMap: removing container" podUID="dcf6c38b-c496-45e8-bd2b-20a83cc67f36" containerName="revoke-keys" Nov 02 03:55:43.568560 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568559 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf6c38b-c496-45e8-bd2b-20a83cc67f36" containerName="revoke-keys" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568574 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568580 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568597 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568603 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568617 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3a47a048-3cff-47cd-b470-cb574ea0670b" containerName="main" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568622 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a47a048-3cff-47cd-b470-cb574ea0670b" containerName="main" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568634 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" Nov 02 03:55:43.568638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568638 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" Nov 02 03:55:43.568876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568714 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" Nov 02 03:55:43.568876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568724 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a47a048-3cff-47cd-b470-cb574ea0670b" containerName="main" Nov 02 03:55:43.568876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568732 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" containerName="istio-proxy" Nov 02 03:55:43.568876 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.568741 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf6c38b-c496-45e8-bd2b-20a83cc67f36" containerName="revoke-keys" Nov 02 03:55:43.573228 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.573209 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.626263 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.626233 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-00359a\"/\"noten-model-00359a-kserve-self-signed-certs\"" Nov 02 03:55:43.626500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.626312 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-00359a\"/\"openshift-service-ca.crt\"" Nov 02 03:55:43.626500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.626432 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-00359a\"/\"kube-root-ca.crt\"" Nov 02 03:55:43.629112 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.629086 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px"] Nov 02 03:55:43.673176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.673148 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.673317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.673181 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.673317 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.673208 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.673427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.673312 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.673464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.673434 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.673553 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.673534 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhn5\" (UniqueName: \"kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.674649 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.674634 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-00359a\"/\"default-dockercfg-99s2g\"" Nov 02 03:55:43.775102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775069 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775145 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhn5\" (UniqueName: \"kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775232 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775261 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775301 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775343 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775554 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775620 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.775693 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.775690 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.777765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.777735 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.777892 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.777835 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.832240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.832170 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhn5\" (UniqueName: \"kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5\") pod \"noten-model-00359a-kserve-5f88c9cfc7-674px\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:43.885598 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:43.885560 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:44.116515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:44.116486 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" path="/var/lib/kubelet/pods/7d5c9167-8e93-4fdc-871d-2d160226364d/volumes" Nov 02 03:55:44.117028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:44.117014 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa41d8c-9720-4f2e-8b2e-d5891138c9f2" path="/var/lib/kubelet/pods/9fa41d8c-9720-4f2e-8b2e-d5891138c9f2/volumes" Nov 02 03:55:44.271810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:44.271785 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px"] Nov 02 03:55:44.273872 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:55:44.273841 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod190e3d2a_47b4_4a92_a365_3699f914a135.slice/crio-10b4a696ce7a579dca0b1dd87831cfbba0715f122c5d983cb8906349f933e39b WatchSource:0}: Error finding container 10b4a696ce7a579dca0b1dd87831cfbba0715f122c5d983cb8906349f933e39b: Status 404 returned error can't find the container with id 10b4a696ce7a579dca0b1dd87831cfbba0715f122c5d983cb8906349f933e39b Nov 02 03:55:44.930144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:44.930106 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" event={"ID":"190e3d2a-47b4-4a92-a365-3699f914a135","Type":"ContainerStarted","Data":"e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5"} Nov 02 03:55:44.930144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:44.930149 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" event={"ID":"190e3d2a-47b4-4a92-a365-3699f914a135","Type":"ContainerStarted","Data":"10b4a696ce7a579dca0b1dd87831cfbba0715f122c5d983cb8906349f933e39b"} Nov 02 03:55:45.046052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:45.046001 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" podStartSLOduration=2.04598707 podStartE2EDuration="2.04598707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:55:45.044530305 +0000 UTC m=+4087.040231249" watchObservedRunningTime="2026-11-02 03:55:45.04598707 +0000 UTC m=+4087.041688012" Nov 02 03:55:45.886185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:45.886151 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:53.886582 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:53.886497 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:53.898974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:53.898945 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:55:54.000805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:55:54.000766 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:56:03.445690 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:56:03.445655 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:56:25.749399 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.749365 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b"] Nov 02 03:56:25.749927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.749911 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5c9167-8e93-4fdc-871d-2d160226364d" containerName="istio-proxy" Nov 02 03:56:25.753218 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.753197 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.809025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.808929 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b"] Nov 02 03:56:25.831815 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.831787 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.831972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.831849 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.831972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.831958 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlms6\" (UniqueName: \"kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.854643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.854618 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:56:25.854770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.854652 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:56:25.933603 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.933571 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.933747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.933632 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.933747 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.933658 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlms6\" (UniqueName: \"kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.934309 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.934283 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.935924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.935898 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:25.991188 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:25.991159 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlms6\" (UniqueName: \"kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6\") pod \"maas-api-revoke-sub-087fcdb0-v495b\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:26.066245 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:26.066182 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:26.438055 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:26.438014 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b"] Nov 02 03:56:26.439977 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:56:26.439950 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327a2ea6_62db_450c_8926_d9371bd0e97d.slice/crio-50599835e0958d1517a503c1a75eddef22fee5234fa08ad9687919c807696e22 WatchSource:0}: Error finding container 50599835e0958d1517a503c1a75eddef22fee5234fa08ad9687919c807696e22: Status 404 returned error can't find the container with id 50599835e0958d1517a503c1a75eddef22fee5234fa08ad9687919c807696e22 Nov 02 03:56:26.646455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:26.646427 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px"] Nov 02 03:56:27.182835 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:27.182799 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" event={"ID":"327a2ea6-62db-450c-8926-d9371bd0e97d","Type":"ContainerStarted","Data":"5b19d25ae30fe96e539c8318d413cc7d7a3f6dcd8dcaf452186ad4e00ac597a1"} Nov 02 03:56:27.183215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:27.182849 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" event={"ID":"327a2ea6-62db-450c-8926-d9371bd0e97d","Type":"ContainerStarted","Data":"50599835e0958d1517a503c1a75eddef22fee5234fa08ad9687919c807696e22"} Nov 02 03:56:27.295230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:27.295185 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" podStartSLOduration=2.295172364 podStartE2EDuration="2.295172364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:56:27.294830042 +0000 UTC m=+4129.290530987" watchObservedRunningTime="2026-11-02 03:56:27.295172364 +0000 UTC m=+4129.290873306" Nov 02 03:56:28.190643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:28.190610 4868 generic.go:356] "Generic (PLEG): container finished" podID="327a2ea6-62db-450c-8926-d9371bd0e97d" containerID="5b19d25ae30fe96e539c8318d413cc7d7a3f6dcd8dcaf452186ad4e00ac597a1" exitCode=0 Nov 02 03:56:28.191041 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:28.190694 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" event={"ID":"327a2ea6-62db-450c-8926-d9371bd0e97d","Type":"ContainerDied","Data":"5b19d25ae30fe96e539c8318d413cc7d7a3f6dcd8dcaf452186ad4e00ac597a1"} Nov 02 03:56:29.330770 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.330742 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:29.372049 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.372016 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca\") pod \"327a2ea6-62db-450c-8926-d9371bd0e97d\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " Nov 02 03:56:29.372213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.372161 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca\") pod \"327a2ea6-62db-450c-8926-d9371bd0e97d\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " Nov 02 03:56:29.372213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.372201 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6\" (UniqueName: \"kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6\") pod \"327a2ea6-62db-450c-8926-d9371bd0e97d\" (UID: \"327a2ea6-62db-450c-8926-d9371bd0e97d\") " Nov 02 03:56:29.372455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.372432 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca" pod "327a2ea6-62db-450c-8926-d9371bd0e97d" (UID: "327a2ea6-62db-450c-8926-d9371bd0e97d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:56:29.372681 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.372663 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:29.374301 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.374280 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6" pod "327a2ea6-62db-450c-8926-d9371bd0e97d" (UID: "327a2ea6-62db-450c-8926-d9371bd0e97d"). InnerVolumeSpecName "kube-api-access-qlms6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:56:29.374417 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.374299 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca" pod "327a2ea6-62db-450c-8926-d9371bd0e97d" (UID: "327a2ea6-62db-450c-8926-d9371bd0e97d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:56:29.473972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.473886 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327a2ea6-62db-450c-8926-d9371bd0e97d-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:29.473972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:29.473921 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlms6\" (UniqueName: \"kubernetes.io/projected/327a2ea6-62db-450c-8926-d9371bd0e97d-kube-api-access-qlms6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:30.203685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:30.203655 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" Nov 02 03:56:30.203819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:30.203654 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b" event={"ID":"327a2ea6-62db-450c-8926-d9371bd0e97d","Type":"ContainerDied","Data":"50599835e0958d1517a503c1a75eddef22fee5234fa08ad9687919c807696e22"} Nov 02 03:56:30.203819 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:30.203776 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50599835e0958d1517a503c1a75eddef22fee5234fa08ad9687919c807696e22" Nov 02 03:56:33.445802 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:56:33.445767 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:56:36.492426 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.492388 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg"] Nov 02 03:56:36.492891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.492878 4868 cpu_manager.go:414] "containerMap: removing container" podUID="327a2ea6-62db-450c-8926-d9371bd0e97d" containerName="revoke-keys" Nov 02 03:56:36.492939 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.492892 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327a2ea6-62db-450c-8926-d9371bd0e97d" containerName="revoke-keys" Nov 02 03:56:36.493015 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.493004 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="327a2ea6-62db-450c-8926-d9371bd0e97d" containerName="revoke-keys" Nov 02 03:56:36.496321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.496303 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.548643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.548537 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.555955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.555914 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.556152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.556067 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwp4\" (UniqueName: \"kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.559494 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.559466 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg"] Nov 02 03:56:36.598704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.598679 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:56:36.598898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.598879 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:56:36.657653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.657615 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwp4\" (UniqueName: \"kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.657811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.657749 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.657811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.657779 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.658420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.658400 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.660142 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.660120 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.713854 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.713806 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwp4\" (UniqueName: \"kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4\") pod \"maas-api-revoke-sub-3efd0c24-jg2kg\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:36.810196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:36.810129 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:37.184122 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:37.184102 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg"] Nov 02 03:56:37.186479 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:56:37.186433 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15ff5ed_859b_4e3a_b1b4_89af12ff8ac6.slice/crio-841a9e984f89b71db61ca22c1131fc8fbecab913294cb37ea2ac0d0e653d7fca WatchSource:0}: Error finding container 841a9e984f89b71db61ca22c1131fc8fbecab913294cb37ea2ac0d0e653d7fca: Status 404 returned error can't find the container with id 841a9e984f89b71db61ca22c1131fc8fbecab913294cb37ea2ac0d0e653d7fca Nov 02 03:56:37.244470 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:37.244444 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" event={"ID":"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6","Type":"ContainerStarted","Data":"841a9e984f89b71db61ca22c1131fc8fbecab913294cb37ea2ac0d0e653d7fca"} Nov 02 03:56:38.261631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:38.261536 4868 generic.go:356] "Generic (PLEG): container finished" podID="d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" containerID="751b2fc7d36161168eb0eb07797f6ae98be84cf947f340dbca4723f4586c2eaa" exitCode=0 Nov 02 03:56:38.262052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:38.261618 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" event={"ID":"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6","Type":"ContainerDied","Data":"751b2fc7d36161168eb0eb07797f6ae98be84cf947f340dbca4723f4586c2eaa"} Nov 02 03:56:39.410157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.410128 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:39.489327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.489300 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca\") pod \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " Nov 02 03:56:39.489500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.489344 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca\") pod \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " Nov 02 03:56:39.489544 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.489496 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4\" (UniqueName: \"kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4\") pod \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\" (UID: \"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6\") " Nov 02 03:56:39.489781 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.489751 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca" pod "d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" (UID: "d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:56:39.489977 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.489957 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:39.491420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.491395 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca" pod "d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" (UID: "d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:56:39.491571 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.491547 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4" pod "d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" (UID: "d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6"). InnerVolumeSpecName "kube-api-access-zcwp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:56:39.590896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.590829 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:39.590896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:39.590854 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwp4\" (UniqueName: \"kubernetes.io/projected/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6-kube-api-access-zcwp4\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:40.274077 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:40.274049 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" Nov 02 03:56:40.274235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:40.274089 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg" event={"ID":"d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6","Type":"ContainerDied","Data":"841a9e984f89b71db61ca22c1131fc8fbecab913294cb37ea2ac0d0e653d7fca"} Nov 02 03:56:40.274235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:40.274141 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="841a9e984f89b71db61ca22c1131fc8fbecab913294cb37ea2ac0d0e653d7fca" Nov 02 03:56:41.747009 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:41.746971 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" podUID="190e3d2a-47b4-4a92-a365-3699f914a135" containerName="main" containerID="cri-o://e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5" gracePeriod=45 Nov 02 03:56:42.007780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.007722 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:56:42.118601 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118574 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs\" (UniqueName: \"kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs\") pod \"190e3d2a-47b4-4a92-a365-3699f914a135\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " Nov 02 03:56:42.118761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118642 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home\") pod \"190e3d2a-47b4-4a92-a365-3699f914a135\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " Nov 02 03:56:42.118761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118680 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir\") pod \"190e3d2a-47b4-4a92-a365-3699f914a135\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " Nov 02 03:56:42.118761 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118706 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm\") pod \"190e3d2a-47b4-4a92-a365-3699f914a135\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " Nov 02 03:56:42.118988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118762 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache\") pod \"190e3d2a-47b4-4a92-a365-3699f914a135\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " Nov 02 03:56:42.118988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118835 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5\" (UniqueName: \"kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5\") pod \"190e3d2a-47b4-4a92-a365-3699f914a135\" (UID: \"190e3d2a-47b4-4a92-a365-3699f914a135\") " Nov 02 03:56:42.118988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118934 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home" pod "190e3d2a-47b4-4a92-a365-3699f914a135" (UID: "190e3d2a-47b4-4a92-a365-3699f914a135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:56:42.118988 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.118951 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir" pod "190e3d2a-47b4-4a92-a365-3699f914a135" (UID: "190e3d2a-47b4-4a92-a365-3699f914a135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:56:42.119200 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.119086 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache" pod "190e3d2a-47b4-4a92-a365-3699f914a135" (UID: "190e3d2a-47b4-4a92-a365-3699f914a135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:56:42.119261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.119209 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:42.119261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.119222 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:42.119261 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.119231 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:42.124656 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.124630 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm" pod "190e3d2a-47b4-4a92-a365-3699f914a135" (UID: "190e3d2a-47b4-4a92-a365-3699f914a135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:56:42.124867 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.124848 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5" pod "190e3d2a-47b4-4a92-a365-3699f914a135" (UID: "190e3d2a-47b4-4a92-a365-3699f914a135"). InnerVolumeSpecName "kube-api-access-dbhn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:56:42.127475 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.127447 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs" pod "190e3d2a-47b4-4a92-a365-3699f914a135" (UID: "190e3d2a-47b4-4a92-a365-3699f914a135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:56:42.220365 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.220334 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190e3d2a-47b4-4a92-a365-3699f914a135-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:42.220509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.220379 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190e3d2a-47b4-4a92-a365-3699f914a135-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:42.220509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.220391 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbhn5\" (UniqueName: \"kubernetes.io/projected/190e3d2a-47b4-4a92-a365-3699f914a135-kube-api-access-dbhn5\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:42.293371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.293265 4868 generic.go:356] "Generic (PLEG): container finished" podID="190e3d2a-47b4-4a92-a365-3699f914a135" containerID="e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5" exitCode=0 Nov 02 03:56:42.293371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.293308 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" Nov 02 03:56:42.293371 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.293341 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" event={"ID":"190e3d2a-47b4-4a92-a365-3699f914a135","Type":"ContainerDied","Data":"e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5"} Nov 02 03:56:42.293593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.293390 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" event={"ID":"190e3d2a-47b4-4a92-a365-3699f914a135","Type":"ContainerDied","Data":"10b4a696ce7a579dca0b1dd87831cfbba0715f122c5d983cb8906349f933e39b"} Nov 02 03:56:42.293593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.293414 4868 scope.go:122] "RemoveContainer" containerID="e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5" Nov 02 03:56:42.303434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.303416 4868 scope.go:122] "RemoveContainer" containerID="e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5" Nov 02 03:56:42.303718 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:56:42.303693 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5\": container with ID starting with e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5 not found: ID does not exist" containerID="e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5" Nov 02 03:56:42.303767 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.303728 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5"} err="failed to get container status \"e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5\": rpc error: code = NotFound desc = could not find container \"e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5\": container with ID starting with e49cfea9e9cf12eb94e73dc5c8aa1a07ad9e4b4267f42a3dfec97622422b0be5 not found: ID does not exist" Nov 02 03:56:42.456523 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.456492 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px"] Nov 02 03:56:42.461922 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.461897 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px"] Nov 02 03:56:42.990589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:42.990549 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-00359a/noten-model-00359a-kserve-5f88c9cfc7-674px" podUID="190e3d2a-47b4-4a92-a365-3699f914a135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 03:56:44.115374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:44.115328 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190e3d2a-47b4-4a92-a365-3699f914a135" path="/var/lib/kubelet/pods/190e3d2a-47b4-4a92-a365-3699f914a135/volumes" Nov 02 03:56:53.067324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067290 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k"] Nov 02 03:56:53.069670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067759 4868 cpu_manager.go:414] "containerMap: removing container" podUID="190e3d2a-47b4-4a92-a365-3699f914a135" containerName="main" Nov 02 03:56:53.069670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067773 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190e3d2a-47b4-4a92-a365-3699f914a135" containerName="main" Nov 02 03:56:53.069670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067805 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" containerName="revoke-keys" Nov 02 03:56:53.069670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067811 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" containerName="revoke-keys" Nov 02 03:56:53.069670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067890 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" containerName="revoke-keys" Nov 02 03:56:53.069670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.067902 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="190e3d2a-47b4-4a92-a365-3699f914a135" containerName="main" Nov 02 03:56:53.071008 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.070992 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.128202 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.128160 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k"] Nov 02 03:56:53.146556 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.146495 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.146704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.146685 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5s6\" (UniqueName: \"kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.146803 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.146788 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.174070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.174044 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:56:53.174070 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.174051 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:56:53.248176 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.248148 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.248330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.248238 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.248330 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.248298 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5s6\" (UniqueName: \"kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.248847 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.248824 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.250589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.250560 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.304626 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.304594 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5s6\" (UniqueName: \"kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6\") pod \"maas-api-revoke-sub-a07c6c42-t7f6k\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.385315 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.385286 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:53.753924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:53.753896 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k"] Nov 02 03:56:53.756437 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:56:53.756401 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648ff475_09ed_4495_a58c_87914523280b.slice/crio-55b59114fbd8368dac5cc28f51f64fecd2f156056e22f8ff281d0168af88ad56 WatchSource:0}: Error finding container 55b59114fbd8368dac5cc28f51f64fecd2f156056e22f8ff281d0168af88ad56: Status 404 returned error can't find the container with id 55b59114fbd8368dac5cc28f51f64fecd2f156056e22f8ff281d0168af88ad56 Nov 02 03:56:54.366171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:54.366127 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" event={"ID":"648ff475-09ed-4495-a58c-87914523280b","Type":"ContainerStarted","Data":"709dd89a0df07941901f067bce8465aea670e6431ca035ceeaa07aa9a2154dd9"} Nov 02 03:56:54.366666 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:54.366181 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" event={"ID":"648ff475-09ed-4495-a58c-87914523280b","Type":"ContainerStarted","Data":"55b59114fbd8368dac5cc28f51f64fecd2f156056e22f8ff281d0168af88ad56"} Nov 02 03:56:54.480267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:54.480213 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" podStartSLOduration=1.48019845 podStartE2EDuration="1.48019845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:56:54.477964697 +0000 UTC m=+4156.473665639" watchObservedRunningTime="2026-11-02 03:56:54.48019845 +0000 UTC m=+4156.475899393" Nov 02 03:56:55.373248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:55.373212 4868 generic.go:356] "Generic (PLEG): container finished" podID="648ff475-09ed-4495-a58c-87914523280b" containerID="709dd89a0df07941901f067bce8465aea670e6431ca035ceeaa07aa9a2154dd9" exitCode=0 Nov 02 03:56:55.373672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:55.373296 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" event={"ID":"648ff475-09ed-4495-a58c-87914523280b","Type":"ContainerDied","Data":"709dd89a0df07941901f067bce8465aea670e6431ca035ceeaa07aa9a2154dd9"} Nov 02 03:56:56.520167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.520143 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:56.588627 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.588599 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6\" (UniqueName: \"kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6\") pod \"648ff475-09ed-4495-a58c-87914523280b\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " Nov 02 03:56:56.588769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.588643 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca\") pod \"648ff475-09ed-4495-a58c-87914523280b\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " Nov 02 03:56:56.588769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.588702 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca\") pod \"648ff475-09ed-4495-a58c-87914523280b\" (UID: \"648ff475-09ed-4495-a58c-87914523280b\") " Nov 02 03:56:56.589074 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.589047 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca" pod "648ff475-09ed-4495-a58c-87914523280b" (UID: "648ff475-09ed-4495-a58c-87914523280b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:56:56.590874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.590853 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca" pod "648ff475-09ed-4495-a58c-87914523280b" (UID: "648ff475-09ed-4495-a58c-87914523280b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:56:56.590874 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.590863 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6" pod "648ff475-09ed-4495-a58c-87914523280b" (UID: "648ff475-09ed-4495-a58c-87914523280b"). InnerVolumeSpecName "kube-api-access-qg5s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:56:56.690248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.690166 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg5s6\" (UniqueName: \"kubernetes.io/projected/648ff475-09ed-4495-a58c-87914523280b-kube-api-access-qg5s6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:56.690248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.690202 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648ff475-09ed-4495-a58c-87914523280b-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:56.690248 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:56.690220 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648ff475-09ed-4495-a58c-87914523280b-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:56:57.386429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:57.386394 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" Nov 02 03:56:57.386429 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:57.386411 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k" event={"ID":"648ff475-09ed-4495-a58c-87914523280b","Type":"ContainerDied","Data":"55b59114fbd8368dac5cc28f51f64fecd2f156056e22f8ff281d0168af88ad56"} Nov 02 03:56:57.386634 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:56:57.386469 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55b59114fbd8368dac5cc28f51f64fecd2f156056e22f8ff281d0168af88ad56" Nov 02 03:57:03.446018 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:03.445982 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:57:03.818442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.818366 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5"] Nov 02 03:57:03.818851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.818838 4868 cpu_manager.go:414] "containerMap: removing container" podUID="648ff475-09ed-4495-a58c-87914523280b" containerName="revoke-keys" Nov 02 03:57:03.818905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.818852 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648ff475-09ed-4495-a58c-87914523280b" containerName="revoke-keys" Nov 02 03:57:03.818976 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.818967 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="648ff475-09ed-4495-a58c-87914523280b" containerName="revoke-keys" Nov 02 03:57:03.822509 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.822492 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.862870 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.862843 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfn9l\" (UniqueName: \"kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.862982 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.862903 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.863050 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.863028 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.875501 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.875477 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:57:03.925086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.925057 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:57:03.927674 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.927656 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5"] Nov 02 03:57:03.929842 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.929824 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11259-30a5ee"] Nov 02 03:57:03.933743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.933725 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:03.964046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.964025 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.964155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.964106 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfn9l\" (UniqueName: \"kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.964199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.964155 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.964677 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.964657 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.966526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.966502 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:03.983435 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.983407 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-30a5ee"] Nov 02 03:57:03.992140 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:03.992121 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t4r8t\"" Nov 02 03:57:04.020670 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.020646 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfn9l\" (UniqueName: \"kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l\") pod \"maas-api-revoke-sub-96225e55-5ggm5\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:04.064993 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.064971 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrfv\" (UniqueName: \"kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv\") pod \"test-curl-11259-30a5ee\" (UID: \"9a8657cc-356b-4b46-8cf5-53387bc87fce\") " pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:04.136088 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.136065 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:04.166166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.166123 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrfv\" (UniqueName: \"kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv\") pod \"test-curl-11259-30a5ee\" (UID: \"9a8657cc-356b-4b46-8cf5-53387bc87fce\") " pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:04.223507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.223438 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrfv\" (UniqueName: \"kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv\") pod \"test-curl-11259-30a5ee\" (UID: \"9a8657cc-356b-4b46-8cf5-53387bc87fce\") " pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:04.245260 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.245230 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:04.506711 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.506685 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5"] Nov 02 03:57:04.509217 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:04.509181 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2aaf09_4a9f_4e51_999e_2bb066f7190f.slice/crio-f517b58ee4b809c21fee3f4ff6f8ad13f9e055380920242ef3b23cee3094a3ed WatchSource:0}: Error finding container f517b58ee4b809c21fee3f4ff6f8ad13f9e055380920242ef3b23cee3094a3ed: Status 404 returned error can't find the container with id f517b58ee4b809c21fee3f4ff6f8ad13f9e055380920242ef3b23cee3094a3ed Nov 02 03:57:04.614469 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.614443 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-30a5ee"] Nov 02 03:57:04.615765 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:04.615740 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8657cc_356b_4b46_8cf5_53387bc87fce.slice/crio-8b54c3c110a95e68ab6947401634d0dd693b0db9fc0d6bfdd64d3e8190a2aecf WatchSource:0}: Error finding container 8b54c3c110a95e68ab6947401634d0dd693b0db9fc0d6bfdd64d3e8190a2aecf: Status 404 returned error can't find the container with id 8b54c3c110a95e68ab6947401634d0dd693b0db9fc0d6bfdd64d3e8190a2aecf Nov 02 03:57:04.617584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:04.617567 4868 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 03:57:05.433853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:05.433815 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-30a5ee" event={"ID":"9a8657cc-356b-4b46-8cf5-53387bc87fce","Type":"ContainerStarted","Data":"8b54c3c110a95e68ab6947401634d0dd693b0db9fc0d6bfdd64d3e8190a2aecf"} Nov 02 03:57:05.435597 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:05.435569 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" event={"ID":"7a2aaf09-4a9f-4e51-999e-2bb066f7190f","Type":"ContainerStarted","Data":"4d872f2e54c42b77a52d79ab1e8cded314ba5376c7b894e3d51a9b741176aea3"} Nov 02 03:57:05.435753 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:05.435612 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" event={"ID":"7a2aaf09-4a9f-4e51-999e-2bb066f7190f","Type":"ContainerStarted","Data":"f517b58ee4b809c21fee3f4ff6f8ad13f9e055380920242ef3b23cee3094a3ed"} Nov 02 03:57:05.547893 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:05.547831 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" podStartSLOduration=2.547813246 podStartE2EDuration="2.547813246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:05.546799726 +0000 UTC m=+4167.542500668" watchObservedRunningTime="2026-11-02 03:57:05.547813246 +0000 UTC m=+4167.543514190" Nov 02 03:57:06.444584 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:06.444544 4868 generic.go:356] "Generic (PLEG): container finished" podID="7a2aaf09-4a9f-4e51-999e-2bb066f7190f" containerID="4d872f2e54c42b77a52d79ab1e8cded314ba5376c7b894e3d51a9b741176aea3" exitCode=0 Nov 02 03:57:06.444783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:06.444632 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" event={"ID":"7a2aaf09-4a9f-4e51-999e-2bb066f7190f","Type":"ContainerDied","Data":"4d872f2e54c42b77a52d79ab1e8cded314ba5376c7b894e3d51a9b741176aea3"} Nov 02 03:57:07.452862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.452830 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-30a5ee" event={"ID":"9a8657cc-356b-4b46-8cf5-53387bc87fce","Type":"ContainerStarted","Data":"7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036"} Nov 02 03:57:07.569102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.569057 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11259-30a5ee" podStartSLOduration=1.825095445 podStartE2EDuration="4.569041985s" totalImagesPullingTime="2.74394654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:07.566665177 +0000 UTC m=+4169.562366122" watchObservedRunningTime="2026-11-02 03:57:07.569041985 +0000 UTC m=+4169.564742927" Nov 02 03:57:07.587851 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.587826 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:07.706078 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.705989 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca\") pod \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " Nov 02 03:57:07.706238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.706109 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca\") pod \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " Nov 02 03:57:07.706238 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.706141 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l\" (UniqueName: \"kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l\") pod \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\" (UID: \"7a2aaf09-4a9f-4e51-999e-2bb066f7190f\") " Nov 02 03:57:07.706583 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.706556 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca" pod "7a2aaf09-4a9f-4e51-999e-2bb066f7190f" (UID: "7a2aaf09-4a9f-4e51-999e-2bb066f7190f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:57:07.708548 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.708483 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca" pod "7a2aaf09-4a9f-4e51-999e-2bb066f7190f" (UID: "7a2aaf09-4a9f-4e51-999e-2bb066f7190f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:57:07.708668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.708602 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l" pod "7a2aaf09-4a9f-4e51-999e-2bb066f7190f" (UID: "7a2aaf09-4a9f-4e51-999e-2bb066f7190f"). InnerVolumeSpecName "kube-api-access-kfn9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:07.807839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.807811 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:07.807839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.807840 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:07.807839 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:07.807850 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfn9l\" (UniqueName: \"kubernetes.io/projected/7a2aaf09-4a9f-4e51-999e-2bb066f7190f-kube-api-access-kfn9l\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:08.460344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:08.460255 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" Nov 02 03:57:08.460344 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:08.460292 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5" event={"ID":"7a2aaf09-4a9f-4e51-999e-2bb066f7190f","Type":"ContainerDied","Data":"f517b58ee4b809c21fee3f4ff6f8ad13f9e055380920242ef3b23cee3094a3ed"} Nov 02 03:57:08.460780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:08.460345 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f517b58ee4b809c21fee3f4ff6f8ad13f9e055380920242ef3b23cee3094a3ed" Nov 02 03:57:09.091306 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.091273 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11259-30a5ee"] Nov 02 03:57:09.094096 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.094073 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11259-30a5ee"] Nov 02 03:57:09.450992 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.450952 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11259-18b6ec"] Nov 02 03:57:09.451534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.451512 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7a2aaf09-4a9f-4e51-999e-2bb066f7190f" containerName="revoke-keys" Nov 02 03:57:09.451651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.451534 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2aaf09-4a9f-4e51-999e-2bb066f7190f" containerName="revoke-keys" Nov 02 03:57:09.451651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.451594 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9a8657cc-356b-4b46-8cf5-53387bc87fce" containerName="test-curl-11259-30a5ee" Nov 02 03:57:09.451651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.451603 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8657cc-356b-4b46-8cf5-53387bc87fce" containerName="test-curl-11259-30a5ee" Nov 02 03:57:09.451814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.451728 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8657cc-356b-4b46-8cf5-53387bc87fce" containerName="test-curl-11259-30a5ee" Nov 02 03:57:09.451814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.451751 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2aaf09-4a9f-4e51-999e-2bb066f7190f" containerName="revoke-keys" Nov 02 03:57:09.455080 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.455057 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:09.465950 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.465921 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11259-30a5ee" podUID="9a8657cc-356b-4b46-8cf5-53387bc87fce" containerName="test-curl-11259-30a5ee" containerID="cri-o://7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036" gracePeriod=2 Nov 02 03:57:09.509298 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.509276 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-18b6ec"] Nov 02 03:57:09.528780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.528750 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5ms2\" (UniqueName: \"kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2\") pod \"test-curl-11259-18b6ec\" (UID: \"f79bcef5-1bd0-4639-ab10-7af467c85378\") " pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:09.615289 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:09.615254 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-30a5ee\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="9a8657cc-356b-4b46-8cf5-53387bc87fce" pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:09.630502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.630473 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ms2\" (UniqueName: \"kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2\") pod \"test-curl-11259-18b6ec\" (UID: \"f79bcef5-1bd0-4639-ab10-7af467c85378\") " pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:09.686750 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.686718 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ms2\" (UniqueName: \"kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2\") pod \"test-curl-11259-18b6ec\" (UID: \"f79bcef5-1bd0-4639-ab10-7af467c85378\") " pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:09.767285 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:09.767226 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:10.139324 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:10.139299 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-18b6ec"] Nov 02 03:57:10.141306 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:10.141275 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf79bcef5_1bd0_4639_ab10_7af467c85378.slice/crio-0af4f78f158660286ea8f953346e6c34010b14c872c5d1f76d8097301b2f1b89 WatchSource:0}: Error finding container 0af4f78f158660286ea8f953346e6c34010b14c872c5d1f76d8097301b2f1b89: Status 404 returned error can't find the container with id 0af4f78f158660286ea8f953346e6c34010b14c872c5d1f76d8097301b2f1b89 Nov 02 03:57:10.473151 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:10.473061 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-18b6ec" event={"ID":"f79bcef5-1bd0-4639-ab10-7af467c85378","Type":"ContainerStarted","Data":"863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119"} Nov 02 03:57:10.473151 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:10.473114 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-18b6ec" event={"ID":"f79bcef5-1bd0-4639-ab10-7af467c85378","Type":"ContainerStarted","Data":"0af4f78f158660286ea8f953346e6c34010b14c872c5d1f76d8097301b2f1b89"} Nov 02 03:57:10.587699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:10.587633 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11259-18b6ec" podStartSLOduration=1.587615713 podStartE2EDuration="1.587615713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:10.58388616 +0000 UTC m=+4172.579587104" watchObservedRunningTime="2026-11-02 03:57:10.587615713 +0000 UTC m=+4172.583316657" Nov 02 03:57:11.697412 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:11.697384 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:11.855619 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:11.855594 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv\" (UniqueName: \"kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv\") pod \"9a8657cc-356b-4b46-8cf5-53387bc87fce\" (UID: \"9a8657cc-356b-4b46-8cf5-53387bc87fce\") " Nov 02 03:57:11.857741 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:11.857714 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv" pod "9a8657cc-356b-4b46-8cf5-53387bc87fce" (UID: "9a8657cc-356b-4b46-8cf5-53387bc87fce"). InnerVolumeSpecName "kube-api-access-zrrfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:11.897037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:11.897013 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11259-18b6ec"] Nov 02 03:57:11.901646 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:11.901620 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11259-18b6ec"] Nov 02 03:57:11.957124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:11.957101 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrrfv\" (UniqueName: \"kubernetes.io/projected/9a8657cc-356b-4b46-8cf5-53387bc87fce-kube-api-access-zrrfv\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:12.032378 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.032338 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11259-01a2e3"] Nov 02 03:57:12.032887 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.032873 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f79bcef5-1bd0-4639-ab10-7af467c85378" containerName="test-curl-11259-18b6ec" Nov 02 03:57:12.032937 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.032889 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79bcef5-1bd0-4639-ab10-7af467c85378" containerName="test-curl-11259-18b6ec" Nov 02 03:57:12.033000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.032990 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79bcef5-1bd0-4639-ab10-7af467c85378" containerName="test-curl-11259-18b6ec" Nov 02 03:57:12.036406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.036391 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:12.090476 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.090447 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-01a2e3"] Nov 02 03:57:12.116964 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.116901 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8657cc-356b-4b46-8cf5-53387bc87fce" path="/var/lib/kubelet/pods/9a8657cc-356b-4b46-8cf5-53387bc87fce/volumes" Nov 02 03:57:12.159387 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.159335 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kcp\" (UniqueName: \"kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp\") pod \"test-curl-11259-01a2e3\" (UID: \"85299cd1-85c4-4a6f-986d-0f7279b8a58d\") " pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:12.260978 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.260947 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kcp\" (UniqueName: \"kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp\") pod \"test-curl-11259-01a2e3\" (UID: \"85299cd1-85c4-4a6f-986d-0f7279b8a58d\") " pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:12.317549 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.317525 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kcp\" (UniqueName: \"kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp\") pod \"test-curl-11259-01a2e3\" (UID: \"85299cd1-85c4-4a6f-986d-0f7279b8a58d\") " pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:12.349321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.349301 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:12.487672 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.487638 4868 generic.go:356] "Generic (PLEG): container finished" podID="9a8657cc-356b-4b46-8cf5-53387bc87fce" containerID="7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036" exitCode=137 Nov 02 03:57:12.487834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.487680 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-30a5ee" Nov 02 03:57:12.487834 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.487729 4868 scope.go:122] "RemoveContainer" containerID="7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036" Nov 02 03:57:12.487942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.487925 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11259-18b6ec" podUID="f79bcef5-1bd0-4639-ab10-7af467c85378" containerName="test-curl-11259-18b6ec" containerID="cri-o://863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119" gracePeriod=2 Nov 02 03:57:12.496728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.496709 4868 scope.go:122] "RemoveContainer" containerID="7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036" Nov 02 03:57:12.497000 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:12.496976 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036\": container with ID starting with 7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036 not found: ID does not exist" containerID="7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036" Nov 02 03:57:12.497056 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.497012 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036"} err="failed to get container status \"7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036\": rpc error: code = NotFound desc = could not find container \"7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036\": container with ID starting with 7bf74fab0685cbd1e84e2e4c6119ddeb308add1065051b49a548db615caba036 not found: ID does not exist" Nov 02 03:57:12.720013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:12.719985 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-01a2e3"] Nov 02 03:57:12.721005 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:12.720976 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85299cd1_85c4_4a6f_986d_0f7279b8a58d.slice/crio-71ba7fafe49e96ce15320a5d8ea21b78c6ad571bc57b922a0dcc74fb3d29f8b5 WatchSource:0}: Error finding container 71ba7fafe49e96ce15320a5d8ea21b78c6ad571bc57b922a0dcc74fb3d29f8b5: Status 404 returned error can't find the container with id 71ba7fafe49e96ce15320a5d8ea21b78c6ad571bc57b922a0dcc74fb3d29f8b5 Nov 02 03:57:13.497031 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:13.496998 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-01a2e3" event={"ID":"85299cd1-85c4-4a6f-986d-0f7279b8a58d","Type":"ContainerStarted","Data":"c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0"} Nov 02 03:57:13.497210 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:13.497042 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-01a2e3" event={"ID":"85299cd1-85c4-4a6f-986d-0f7279b8a58d","Type":"ContainerStarted","Data":"71ba7fafe49e96ce15320a5d8ea21b78c6ad571bc57b922a0dcc74fb3d29f8b5"} Nov 02 03:57:13.608609 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:13.608561 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11259-01a2e3" podStartSLOduration=2.608546714 podStartE2EDuration="2.608546714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:13.607231018 +0000 UTC m=+4175.602931964" watchObservedRunningTime="2026-11-02 03:57:13.608546714 +0000 UTC m=+4175.604247656" Nov 02 03:57:14.749280 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:14.749215 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:14.891152 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:14.891119 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2\" (UniqueName: \"kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2\") pod \"f79bcef5-1bd0-4639-ab10-7af467c85378\" (UID: \"f79bcef5-1bd0-4639-ab10-7af467c85378\") " Nov 02 03:57:14.893305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:14.893279 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2" pod "f79bcef5-1bd0-4639-ab10-7af467c85378" (UID: "f79bcef5-1bd0-4639-ab10-7af467c85378"). InnerVolumeSpecName "kube-api-access-n5ms2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:14.992833 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:14.992799 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5ms2\" (UniqueName: \"kubernetes.io/projected/f79bcef5-1bd0-4639-ab10-7af467c85378-kube-api-access-n5ms2\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:15.039801 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.039734 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11259-01a2e3"] Nov 02 03:57:15.043367 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.043330 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11259-01a2e3"] Nov 02 03:57:15.171016 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.170980 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11259-e7fc1a"] Nov 02 03:57:15.171517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.171499 4868 cpu_manager.go:414] "containerMap: removing container" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" containerName="test-curl-11259-01a2e3" Nov 02 03:57:15.171517 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.171516 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" containerName="test-curl-11259-01a2e3" Nov 02 03:57:15.171637 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.171601 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" containerName="test-curl-11259-01a2e3" Nov 02 03:57:15.175103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.175081 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:15.229282 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.229253 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-e7fc1a"] Nov 02 03:57:15.296372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.296269 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr2ts\" (UniqueName: \"kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts\") pod \"test-curl-11259-e7fc1a\" (UID: \"51649dbb-a99c-4273-99ec-d6ff77e969dc\") " pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:15.397588 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.397555 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr2ts\" (UniqueName: \"kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts\") pod \"test-curl-11259-e7fc1a\" (UID: \"51649dbb-a99c-4273-99ec-d6ff77e969dc\") " pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:15.453552 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.453524 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr2ts\" (UniqueName: \"kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts\") pod \"test-curl-11259-e7fc1a\" (UID: \"51649dbb-a99c-4273-99ec-d6ff77e969dc\") " pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:15.487208 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.487182 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:15.512849 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.512820 4868 generic.go:356] "Generic (PLEG): container finished" podID="f79bcef5-1bd0-4639-ab10-7af467c85378" containerID="863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119" exitCode=137 Nov 02 03:57:15.512983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.512882 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:15.512983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.512925 4868 scope.go:122] "RemoveContainer" containerID="863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119" Nov 02 03:57:15.513459 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.513289 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11259-01a2e3" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" containerName="test-curl-11259-01a2e3" containerID="cri-o://c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0" gracePeriod=2 Nov 02 03:57:15.528829 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.528807 4868 scope.go:122] "RemoveContainer" containerID="863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119" Nov 02 03:57:15.529100 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:15.529080 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119\": container with ID starting with 863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119 not found: ID does not exist" containerID="863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119" Nov 02 03:57:15.529161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.529110 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119"} err="failed to get container status \"863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119\": rpc error: code = NotFound desc = could not find container \"863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119\": container with ID starting with 863a9ea4af6e24f6b40c81d8aff8be65e836346a485cabe3b41e64d43e623119 not found: ID does not exist" Nov 02 03:57:15.564129 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:15.564091 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-01a2e3\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:15.614551 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:15.614516 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-01a2e3\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:15.664897 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:15.664867 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-18b6ec\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="f79bcef5-1bd0-4639-ab10-7af467c85378" pod="opendatahub/test-curl-11259-18b6ec" Nov 02 03:57:15.859805 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:15.859778 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-e7fc1a"] Nov 02 03:57:15.861659 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:15.861626 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51649dbb_a99c_4273_99ec_d6ff77e969dc.slice/crio-d2386a1823f76a02b77b7595e56a85560876de04bf1b0a1e7ff72315dc80f400 WatchSource:0}: Error finding container d2386a1823f76a02b77b7595e56a85560876de04bf1b0a1e7ff72315dc80f400: Status 404 returned error can't find the container with id d2386a1823f76a02b77b7595e56a85560876de04bf1b0a1e7ff72315dc80f400 Nov 02 03:57:16.116161 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:16.116079 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79bcef5-1bd0-4639-ab10-7af467c85378" path="/var/lib/kubelet/pods/f79bcef5-1bd0-4639-ab10-7af467c85378/volumes" Nov 02 03:57:16.521863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:16.521777 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-e7fc1a" event={"ID":"51649dbb-a99c-4273-99ec-d6ff77e969dc","Type":"ContainerStarted","Data":"8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a"} Nov 02 03:57:16.521863 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:16.521824 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-e7fc1a" event={"ID":"51649dbb-a99c-4273-99ec-d6ff77e969dc","Type":"ContainerStarted","Data":"d2386a1823f76a02b77b7595e56a85560876de04bf1b0a1e7ff72315dc80f400"} Nov 02 03:57:16.634079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:16.634026 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11259-e7fc1a" podStartSLOduration=1.63401247 podStartE2EDuration="1.63401247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:16.631676662 +0000 UTC m=+4178.627377607" watchObservedRunningTime="2026-11-02 03:57:16.63401247 +0000 UTC m=+4178.629713413" Nov 02 03:57:17.755522 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:17.755495 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:17.924547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:17.924519 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp\" (UniqueName: \"kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp\") pod \"85299cd1-85c4-4a6f-986d-0f7279b8a58d\" (UID: \"85299cd1-85c4-4a6f-986d-0f7279b8a58d\") " Nov 02 03:57:17.926580 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:17.926557 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp" pod "85299cd1-85c4-4a6f-986d-0f7279b8a58d" (UID: "85299cd1-85c4-4a6f-986d-0f7279b8a58d"). InnerVolumeSpecName "kube-api-access-77kcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:17.953793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:17.953760 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11259-e7fc1a"] Nov 02 03:57:17.957547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:17.957521 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11259-e7fc1a"] Nov 02 03:57:18.025696 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.025667 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kcp\" (UniqueName: \"kubernetes.io/projected/85299cd1-85c4-4a6f-986d-0f7279b8a58d-kube-api-access-77kcp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:18.096783 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.096756 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11259-62a0bb"] Nov 02 03:57:18.097241 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.097228 4868 cpu_manager.go:414] "containerMap: removing container" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" containerName="test-curl-11259-e7fc1a" Nov 02 03:57:18.097284 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.097242 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" containerName="test-curl-11259-e7fc1a" Nov 02 03:57:18.097347 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.097337 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" containerName="test-curl-11259-e7fc1a" Nov 02 03:57:18.100731 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.100715 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:18.115159 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.115136 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" path="/var/lib/kubelet/pods/85299cd1-85c4-4a6f-986d-0f7279b8a58d/volumes" Nov 02 03:57:18.152919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.152894 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-62a0bb"] Nov 02 03:57:18.228570 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.228499 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gmn\" (UniqueName: \"kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn\") pod \"test-curl-11259-62a0bb\" (UID: \"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1\") " pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:18.330216 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.330187 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gmn\" (UniqueName: \"kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn\") pod \"test-curl-11259-62a0bb\" (UID: \"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1\") " pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:18.386282 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.386253 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gmn\" (UniqueName: \"kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn\") pod \"test-curl-11259-62a0bb\" (UID: \"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1\") " pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:18.412305 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.412279 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:18.535879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.535807 4868 generic.go:356] "Generic (PLEG): container finished" podID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" containerID="c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0" exitCode=137 Nov 02 03:57:18.535879 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.535852 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:18.536093 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.535906 4868 scope.go:122] "RemoveContainer" containerID="c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0" Nov 02 03:57:18.536171 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.536103 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11259-e7fc1a" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" containerName="test-curl-11259-e7fc1a" containerID="cri-o://8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a" gracePeriod=2 Nov 02 03:57:18.545043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.545027 4868 scope.go:122] "RemoveContainer" containerID="c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0" Nov 02 03:57:18.545325 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:18.545299 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0\": container with ID starting with c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0 not found: ID does not exist" containerID="c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0" Nov 02 03:57:18.545455 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.545330 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0"} err="failed to get container status \"c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0\": rpc error: code = NotFound desc = could not find container \"c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0\": container with ID starting with c116206b261739a057dcf2297e0e93cead5077b3ea2bda16dcf5894d97bac7d0 not found: ID does not exist" Nov 02 03:57:18.586711 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:18.586681 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-01a2e3\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:18.637282 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:18.637256 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-01a2e3\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:18.687719 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:18.687691 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-e7fc1a\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:18.779432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:18.779407 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11259-62a0bb"] Nov 02 03:57:18.781097 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:18.781072 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1ba7bff_d7b3_4476_b30a_3bff1d229ef1.slice/crio-f2a77cb95a46dec94fb2bf49ebe111bcd7ece439d89efbedfc5404da66b70403 WatchSource:0}: Error finding container f2a77cb95a46dec94fb2bf49ebe111bcd7ece439d89efbedfc5404da66b70403: Status 404 returned error can't find the container with id f2a77cb95a46dec94fb2bf49ebe111bcd7ece439d89efbedfc5404da66b70403 Nov 02 03:57:19.543632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:19.543601 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-62a0bb" event={"ID":"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1","Type":"ContainerStarted","Data":"1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6"} Nov 02 03:57:19.543632 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:19.543639 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11259-62a0bb" event={"ID":"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1","Type":"ContainerStarted","Data":"f2a77cb95a46dec94fb2bf49ebe111bcd7ece439d89efbedfc5404da66b70403"} Nov 02 03:57:19.594071 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:19.594039 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-e7fc1a\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:19.771026 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:19.770988 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-01a2e3\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="85299cd1-85c4-4a6f-986d-0f7279b8a58d" pod="opendatahub/test-curl-11259-01a2e3" Nov 02 03:57:20.787427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:20.787402 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:20.838067 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:20.838004 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-e7fc1a\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:20.959571 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:20.959545 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts\" (UniqueName: \"kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts\") pod \"51649dbb-a99c-4273-99ec-d6ff77e969dc\" (UID: \"51649dbb-a99c-4273-99ec-d6ff77e969dc\") " Nov 02 03:57:20.961638 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:20.961612 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts" pod "51649dbb-a99c-4273-99ec-d6ff77e969dc" (UID: "51649dbb-a99c-4273-99ec-d6ff77e969dc"). InnerVolumeSpecName "kube-api-access-nr2ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:21.060502 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.060473 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr2ts\" (UniqueName: \"kubernetes.io/projected/51649dbb-a99c-4273-99ec-d6ff77e969dc-kube-api-access-nr2ts\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:21.151117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.151068 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11259-62a0bb" podStartSLOduration=3.151053008 podStartE2EDuration="3.151053008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:19.720492161 +0000 UTC m=+4181.716193105" watchObservedRunningTime="2026-11-02 03:57:21.151053008 +0000 UTC m=+4183.146753950" Nov 02 03:57:21.154278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.154246 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11259-62a0bb"] Nov 02 03:57:21.160699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.160668 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11259-62a0bb"] Nov 02 03:57:21.557066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.556973 4868 generic.go:356] "Generic (PLEG): container finished" podID="51649dbb-a99c-4273-99ec-d6ff77e969dc" containerID="8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a" exitCode=137 Nov 02 03:57:21.557066 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.557025 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:21.557294 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.557074 4868 scope.go:122] "RemoveContainer" containerID="8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a" Nov 02 03:57:21.557416 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.557388 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11259-62a0bb" podUID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" containerName="test-curl-11259-62a0bb" containerID="cri-o://1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6" gracePeriod=2 Nov 02 03:57:21.567757 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.567735 4868 scope.go:122] "RemoveContainer" containerID="8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a" Nov 02 03:57:21.567993 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:21.567973 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a\": container with ID starting with 8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a not found: ID does not exist" containerID="8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a" Nov 02 03:57:21.568061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:21.568008 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a"} err="failed to get container status \"8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a\": rpc error: code = NotFound desc = could not find container \"8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a\": container with ID starting with 8e694ead50defc486ebecf0eade6aa622fac35ab643c2c7ecc73aa335edb1b1a not found: ID does not exist" Nov 02 03:57:21.607771 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:21.607745 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-e7fc1a\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:21.658200 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:21.658171 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-e7fc1a\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" pod="opendatahub/test-curl-11259-e7fc1a" Nov 02 03:57:21.708519 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:21.708483 4868 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11259-62a0bb\" is forbidden: User \"system:node:ip-10-0-137-74.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-74.us-west-1.compute.internal' and this object" podUID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:22.115017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:22.114982 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51649dbb-a99c-4273-99ec-d6ff77e969dc" path="/var/lib/kubelet/pods/51649dbb-a99c-4273-99ec-d6ff77e969dc/volumes" Nov 02 03:57:23.513236 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.513202 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp"] Nov 02 03:57:23.513699 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.513685 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" containerName="test-curl-11259-62a0bb" Nov 02 03:57:23.513772 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.513700 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" containerName="test-curl-11259-62a0bb" Nov 02 03:57:23.513814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.513808 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" containerName="test-curl-11259-62a0bb" Nov 02 03:57:23.518162 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.518140 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.585793 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.585768 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-61c297-openshift-default-dockercfg-g99rz\"" Nov 02 03:57:23.592318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592287 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592323 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592434 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592361 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtdr\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592481 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592499 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592534 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592617 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592562 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592714 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592614 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.592752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.592734 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.622944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.622616 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp"] Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.694992 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695057 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695094 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtdr\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695172 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695198 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695257 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695299 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695398 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695326 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.695797 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.695460 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.696266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.696234 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.696560 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.696533 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.696974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.696949 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.705831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.697189 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.705831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.697513 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.705831 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.699298 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.723635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.723565 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.756453 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.756424 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.756616 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.756593 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtdr\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr\") pod \"e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.831919 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.831889 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:23.832427 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.832403 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:23.901110 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.901078 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn\" (UniqueName: \"kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn\") pod \"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1\" (UID: \"d1ba7bff-d7b3-4476-b30a-3bff1d229ef1\") " Nov 02 03:57:23.906515 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:23.906477 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn" pod "d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" (UID: "d1ba7bff-d7b3-4476-b30a-3bff1d229ef1"). InnerVolumeSpecName "kube-api-access-j4gmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:24.002449 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.002413 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gmn\" (UniqueName: \"kubernetes.io/projected/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1-kube-api-access-j4gmn\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:24.116028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.115999 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" path="/var/lib/kubelet/pods/d1ba7bff-d7b3-4476-b30a-3bff1d229ef1/volumes" Nov 02 03:57:24.274983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.274953 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp"] Nov 02 03:57:24.275379 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:24.275327 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ae4893_94e4_484c_bac3_c1f582c7d863.slice/crio-5084c169f859cdbdaaeb215e44535370defb5dc605c34717fb1b3682fa96d921 WatchSource:0}: Error finding container 5084c169f859cdbdaaeb215e44535370defb5dc605c34717fb1b3682fa96d921: Status 404 returned error can't find the container with id 5084c169f859cdbdaaeb215e44535370defb5dc605c34717fb1b3682fa96d921 Nov 02 03:57:24.278000 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.277970 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:57:24.278091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.278027 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:57:24.278091 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.278061 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:57:24.581614 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.581575 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" event={"ID":"80ae4893-94e4-484c-bac3-c1f582c7d863","Type":"ContainerStarted","Data":"bc0bbc40c7a039e929b15414f149ce93775b64368895634f041d1ef42d7637e0"} Nov 02 03:57:24.582058 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.581644 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" event={"ID":"80ae4893-94e4-484c-bac3-c1f582c7d863","Type":"ContainerStarted","Data":"5084c169f859cdbdaaeb215e44535370defb5dc605c34717fb1b3682fa96d921"} Nov 02 03:57:24.584183 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.584150 4868 generic.go:356] "Generic (PLEG): container finished" podID="d1ba7bff-d7b3-4476-b30a-3bff1d229ef1" containerID="1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6" exitCode=137 Nov 02 03:57:24.584363 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.584206 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11259-62a0bb" Nov 02 03:57:24.584612 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.584209 4868 scope.go:122] "RemoveContainer" containerID="1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6" Nov 02 03:57:24.593804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.593785 4868 scope.go:122] "RemoveContainer" containerID="1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6" Nov 02 03:57:24.594126 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:24.594098 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6\": container with ID starting with 1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6 not found: ID does not exist" containerID="1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6" Nov 02 03:57:24.594218 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.594138 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6"} err="failed to get container status \"1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6\": rpc error: code = NotFound desc = could not find container \"1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6\": container with ID starting with 1db40e4664da425b7385e986234c1f51d2f05544b72d409aa89f03b4a9b7f9e6 not found: ID does not exist" Nov 02 03:57:24.699921 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.699860 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podStartSLOduration=1.699846459 podStartE2EDuration="1.699846459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:24.697452009 +0000 UTC m=+4186.693152952" watchObservedRunningTime="2026-11-02 03:57:24.699846459 +0000 UTC m=+4186.695547402" Nov 02 03:57:24.839155 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:24.834467 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:25.839076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:25.839041 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:26.596773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:26.596745 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:26.597817 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:26.597797 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:57:28.901037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:28.901000 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5"] Nov 02 03:57:33.445289 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:33.445253 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:57:34.615483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.615447 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75"] Nov 02 03:57:34.620809 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.620786 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.672877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.672846 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-61c297-dockercfg-bf7h2\"" Nov 02 03:57:34.777653 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.777617 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-61c297\"" Nov 02 03:57:34.779264 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.779233 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6stb\" (UniqueName: \"kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb\") pod \"payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.779667 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.779633 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.803871 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.803811 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75"] Nov 02 03:57:34.806307 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.806274 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g"] Nov 02 03:57:34.811524 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.811498 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:34.836148 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.836119 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g"] Nov 02 03:57:34.880769 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.880685 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.880898 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.880813 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6stb\" (UniqueName: \"kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb\") pod \"payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.887321 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.881967 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.937072 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.937036 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6stb\" (UniqueName: \"kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb\") pod \"payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:34.981676 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.981644 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkkc\" (UniqueName: \"kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc\") pod \"payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:34.981850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:34.981827 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:35.083014 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.082971 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkkc\" (UniqueName: \"kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc\") pod \"payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:35.083218 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.083129 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:35.083862 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.083841 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:35.140149 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.140082 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkkc\" (UniqueName: \"kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc\") pod \"payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:35.234558 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.234526 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:35.425896 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.425814 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:35.611296 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.611270 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75"] Nov 02 03:57:35.612925 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:35.612898 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2e4743_1cd0_4914_a9d5_6310a23a86ce.slice/crio-2094f0ccf1cd7627d7ff6743afe4690eb47895c55801a7ac2372a0f4f3832f42 WatchSource:0}: Error finding container 2094f0ccf1cd7627d7ff6743afe4690eb47895c55801a7ac2372a0f4f3832f42: Status 404 returned error can't find the container with id 2094f0ccf1cd7627d7ff6743afe4690eb47895c55801a7ac2372a0f4f3832f42 Nov 02 03:57:35.653530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.652002 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" event={"ID":"7a2e4743-1cd0-4914-a9d5-6310a23a86ce","Type":"ContainerStarted","Data":"2094f0ccf1cd7627d7ff6743afe4690eb47895c55801a7ac2372a0f4f3832f42"} Nov 02 03:57:35.800109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:35.800083 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g"] Nov 02 03:57:35.801288 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:35.801267 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9df8a5_1612_4991_a62d_f8509a92241a.slice/crio-dd7c40ae11648f24e4d7a2498d750ac95e9e9539b0012ec38b1b35790de5877c WatchSource:0}: Error finding container dd7c40ae11648f24e4d7a2498d750ac95e9e9539b0012ec38b1b35790de5877c: Status 404 returned error can't find the container with id dd7c40ae11648f24e4d7a2498d750ac95e9e9539b0012ec38b1b35790de5877c Nov 02 03:57:36.659840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.659807 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" event={"ID":"7a2e4743-1cd0-4914-a9d5-6310a23a86ce","Type":"ContainerStarted","Data":"d0df4bae75813c50df39a44e9a91cc85e7a768c1f4331c455f92062da2f79758"} Nov 02 03:57:36.659840 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.659869 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:36.661406 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.661378 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" event={"ID":"be9df8a5-1612-4991-a62d-f8509a92241a","Type":"ContainerStarted","Data":"7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca"} Nov 02 03:57:36.661559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.661429 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" event={"ID":"be9df8a5-1612-4991-a62d-f8509a92241a","Type":"ContainerStarted","Data":"dd7c40ae11648f24e4d7a2498d750ac95e9e9539b0012ec38b1b35790de5877c"} Nov 02 03:57:36.661559 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.661460 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:36.779334 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.779276 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" podStartSLOduration=2.7792589420000002 podStartE2EDuration="2.779258942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:36.778696117 +0000 UTC m=+4198.774397074" watchObservedRunningTime="2026-11-02 03:57:36.779258942 +0000 UTC m=+4198.774959885" Nov 02 03:57:36.909792 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:36.892108 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" podStartSLOduration=2.892091411 podStartE2EDuration="2.892091411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:36.891673081 +0000 UTC m=+4198.887374380" watchObservedRunningTime="2026-11-02 03:57:36.892091411 +0000 UTC m=+4198.887792357" Nov 02 03:57:38.653717 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:38.653685 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:57:38.664166 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:38.664145 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 03:57:39.434314 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.434270 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9"] Nov 02 03:57:39.442274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.442251 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.510740 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.510706 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9"] Nov 02 03:57:39.581798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.581760 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9b7f43\"" Nov 02 03:57:39.582020 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.581930 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9b7f43-dockercfg-jzs2x\"" Nov 02 03:57:39.582630 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.582575 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.582981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.582948 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w998w\" (UniqueName: \"kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w\") pod \"payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.684500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.684433 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.684884 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.684570 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w998w\" (UniqueName: \"kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w\") pod \"payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.685092 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.685070 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.740764 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.740739 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w998w\" (UniqueName: \"kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w\") pod \"payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:39.756633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:39.756607 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:40.132624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:40.132597 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9"] Nov 02 03:57:40.133853 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:40.133827 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddd750e_10d3_4dc4_a582_308c1e79c3e6.slice/crio-83bc0a731eb00f3448f1f8ebc1b981afea56f3aef0945e4a746b111bc34845ab WatchSource:0}: Error finding container 83bc0a731eb00f3448f1f8ebc1b981afea56f3aef0945e4a746b111bc34845ab: Status 404 returned error can't find the container with id 83bc0a731eb00f3448f1f8ebc1b981afea56f3aef0945e4a746b111bc34845ab Nov 02 03:57:40.688755 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:40.688719 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" event={"ID":"2ddd750e-10d3-4dc4-a582-308c1e79c3e6","Type":"ContainerStarted","Data":"fd8f7d8a9d6c3e63fec6ad0a3beb94b3f35624ad374260f1ff4a4351aa31bb28"} Nov 02 03:57:40.689187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:40.688780 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" event={"ID":"2ddd750e-10d3-4dc4-a582-308c1e79c3e6","Type":"ContainerStarted","Data":"83bc0a731eb00f3448f1f8ebc1b981afea56f3aef0945e4a746b111bc34845ab"} Nov 02 03:57:40.689187 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:40.688810 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:40.802021 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:40.801951 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" podStartSLOduration=1.801932473 podStartE2EDuration="1.801932473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:40.800620172 +0000 UTC m=+4202.796321114" watchObservedRunningTime="2026-11-02 03:57:40.801932473 +0000 UTC m=+4202.797633418" Nov 02 03:57:43.921531 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:43.921478 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" podUID="7aff6c10-c21e-4453-ad0d-8abcdd626812" containerName="main" containerID="cri-o://261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3" gracePeriod=45 Nov 02 03:57:44.184376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.184298 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:57:44.231953 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.231922 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm\") pod \"7aff6c10-c21e-4453-ad0d-8abcdd626812\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " Nov 02 03:57:44.232086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.231970 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88\" (UniqueName: \"kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88\") pod \"7aff6c10-c21e-4453-ad0d-8abcdd626812\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " Nov 02 03:57:44.232086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232005 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs\" (UniqueName: \"kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs\") pod \"7aff6c10-c21e-4453-ad0d-8abcdd626812\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " Nov 02 03:57:44.232086 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232076 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir\") pod \"7aff6c10-c21e-4453-ad0d-8abcdd626812\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " Nov 02 03:57:44.232267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232204 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache\") pod \"7aff6c10-c21e-4453-ad0d-8abcdd626812\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " Nov 02 03:57:44.232267 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232239 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home\") pod \"7aff6c10-c21e-4453-ad0d-8abcdd626812\" (UID: \"7aff6c10-c21e-4453-ad0d-8abcdd626812\") " Nov 02 03:57:44.232402 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232340 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir" pod "7aff6c10-c21e-4453-ad0d-8abcdd626812" (UID: "7aff6c10-c21e-4453-ad0d-8abcdd626812"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:57:44.232530 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232494 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache" pod "7aff6c10-c21e-4453-ad0d-8abcdd626812" (UID: "7aff6c10-c21e-4453-ad0d-8abcdd626812"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:57:44.232635 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232611 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home" pod "7aff6c10-c21e-4453-ad0d-8abcdd626812" (UID: "7aff6c10-c21e-4453-ad0d-8abcdd626812"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:57:44.232763 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232747 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:44.232822 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232768 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:44.232822 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.232784 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:44.234500 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.234478 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm" pod "7aff6c10-c21e-4453-ad0d-8abcdd626812" (UID: "7aff6c10-c21e-4453-ad0d-8abcdd626812"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:57:44.234703 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.234678 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88" pod "7aff6c10-c21e-4453-ad0d-8abcdd626812" (UID: "7aff6c10-c21e-4453-ad0d-8abcdd626812"). InnerVolumeSpecName "kube-api-access-j8p88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:44.234786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.234743 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs" pod "7aff6c10-c21e-4453-ad0d-8abcdd626812" (UID: "7aff6c10-c21e-4453-ad0d-8abcdd626812"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:57:44.334230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.334192 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aff6c10-c21e-4453-ad0d-8abcdd626812-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:44.334230 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.334229 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8p88\" (UniqueName: \"kubernetes.io/projected/7aff6c10-c21e-4453-ad0d-8abcdd626812-kube-api-access-j8p88\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:44.334437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.334246 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aff6c10-c21e-4453-ad0d-8abcdd626812-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:44.719255 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.719216 4868 generic.go:356] "Generic (PLEG): container finished" podID="7aff6c10-c21e-4453-ad0d-8abcdd626812" containerID="261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3" exitCode=0 Nov 02 03:57:44.719473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.719266 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" Nov 02 03:57:44.719473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.719281 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" event={"ID":"7aff6c10-c21e-4453-ad0d-8abcdd626812","Type":"ContainerDied","Data":"261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3"} Nov 02 03:57:44.719473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.719319 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5" event={"ID":"7aff6c10-c21e-4453-ad0d-8abcdd626812","Type":"ContainerDied","Data":"b018307e632225d9183fc9d5a8dfd439a169542bbd66d5cd95919b51f35d1dfb"} Nov 02 03:57:44.719473 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.719341 4868 scope.go:122] "RemoveContainer" containerID="261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3" Nov 02 03:57:44.730207 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.730186 4868 scope.go:122] "RemoveContainer" containerID="261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3" Nov 02 03:57:44.730505 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:57:44.730480 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3\": container with ID starting with 261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3 not found: ID does not exist" containerID="261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3" Nov 02 03:57:44.730579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.730517 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3"} err="failed to get container status \"261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3\": rpc error: code = NotFound desc = could not find container \"261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3\": container with ID starting with 261b5baf1e299615d07f6cc0cb8ad1e53cb7ab822ed1191655cfc50938483ce3 not found: ID does not exist" Nov 02 03:57:44.882076 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.882044 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5"] Nov 02 03:57:44.886732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:44.886707 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a7cf93/auth-test-model-a7cf93-kserve-59687b989c-sffm5"] Nov 02 03:57:46.114780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:46.114750 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aff6c10-c21e-4453-ad0d-8abcdd626812" path="/var/lib/kubelet/pods/7aff6c10-c21e-4453-ad0d-8abcdd626812/volumes" Nov 02 03:57:47.670938 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:47.670902 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 03:57:47.671428 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:47.671090 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 03:57:50.173979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.173946 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v"] Nov 02 03:57:50.174479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.174464 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7aff6c10-c21e-4453-ad0d-8abcdd626812" containerName="main" Nov 02 03:57:50.174535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.174487 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aff6c10-c21e-4453-ad0d-8abcdd626812" containerName="main" Nov 02 03:57:50.174588 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.174579 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aff6c10-c21e-4453-ad0d-8abcdd626812" containerName="main" Nov 02 03:57:50.177673 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.177657 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.229732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.229697 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:57:50.279102 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.279073 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:57:50.281773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.281748 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v"] Nov 02 03:57:50.307144 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.307117 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlbn\" (UniqueName: \"kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.307272 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.307254 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.307382 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.307338 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.408810 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.408781 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlbn\" (UniqueName: \"kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.408965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.408871 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.408965 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.408930 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.409534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.409510 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.411546 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.411523 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.466002 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.465946 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlbn\" (UniqueName: \"kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn\") pod \"maas-api-revoke-sub-6c2bfaea-mnl5v\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.490585 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.490559 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:50.862951 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:50.862920 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v"] Nov 02 03:57:50.864157 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:50.864130 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eea3d2_c0b0_44d4_a7b2_7078d918f9e8.slice/crio-ae862ec3c119a8670f7375f8efac4ab2d402f965518f068e38cf8ba930ea8896 WatchSource:0}: Error finding container ae862ec3c119a8670f7375f8efac4ab2d402f965518f068e38cf8ba930ea8896: Status 404 returned error can't find the container with id ae862ec3c119a8670f7375f8efac4ab2d402f965518f068e38cf8ba930ea8896 Nov 02 03:57:51.701265 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:51.701236 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 03:57:51.771059 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:51.771016 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" event={"ID":"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8","Type":"ContainerStarted","Data":"b88e6ad84d1f86720ae1e2504b8f44c3b987eb9821c4155336e2dddad4624a2d"} Nov 02 03:57:51.771217 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:51.771074 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" event={"ID":"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8","Type":"ContainerStarted","Data":"ae862ec3c119a8670f7375f8efac4ab2d402f965518f068e38cf8ba930ea8896"} Nov 02 03:57:51.929581 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:51.929507 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" podStartSLOduration=1.9294907289999998 podStartE2EDuration="1.929490729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:51.92916366 +0000 UTC m=+4213.924864603" watchObservedRunningTime="2026-11-02 03:57:51.929490729 +0000 UTC m=+4213.925191672" Nov 02 03:57:52.778164 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:52.778128 4868 generic.go:356] "Generic (PLEG): container finished" podID="70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" containerID="b88e6ad84d1f86720ae1e2504b8f44c3b987eb9821c4155336e2dddad4624a2d" exitCode=0 Nov 02 03:57:52.778591 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:52.778215 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" event={"ID":"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8","Type":"ContainerDied","Data":"b88e6ad84d1f86720ae1e2504b8f44c3b987eb9821c4155336e2dddad4624a2d"} Nov 02 03:57:53.496855 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.496823 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5"] Nov 02 03:57:53.500669 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.500649 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.559471 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.559442 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5"] Nov 02 03:57:53.650044 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.650012 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.650215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.650058 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.650215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.650157 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv4cc\" (UniqueName: \"kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.751043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.750963 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.751043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.751040 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv4cc\" (UniqueName: \"kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.751229 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.751196 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.751651 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.751628 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.753543 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.753522 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.807841 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.807812 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv4cc\" (UniqueName: \"kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc\") pod \"maas-api-revoke-sub-3a53e5fb-tcwl5\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.814436 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.814410 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:53.949137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:53.949113 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:54.054920 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.054845 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn\" (UniqueName: \"kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn\") pod \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " Nov 02 03:57:54.055052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.054977 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca\") pod \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " Nov 02 03:57:54.055157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.055138 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca\") pod \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\" (UID: \"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8\") " Nov 02 03:57:54.055527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.055503 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca" pod "70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" (UID: "70eea3d2-c0b0-44d4-a7b2-7078d918f9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:57:54.056999 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.056974 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca" pod "70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" (UID: "70eea3d2-c0b0-44d4-a7b2-7078d918f9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:57:54.057121 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.057078 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn" pod "70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" (UID: "70eea3d2-c0b0-44d4-a7b2-7078d918f9e8"). InnerVolumeSpecName "kube-api-access-vnlbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:54.156569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.156538 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:54.156569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.156562 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnlbn\" (UniqueName: \"kubernetes.io/projected/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-kube-api-access-vnlbn\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:54.156569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.156574 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:54.194995 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.194970 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5"] Nov 02 03:57:54.196378 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:57:54.196325 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9045f2f7_7e64_4a29_b7fa_d4c369063e5b.slice/crio-f6a9ab90643271cc60dac2f8db29f25565d4be781b744292cdf6a861d53497d6 WatchSource:0}: Error finding container f6a9ab90643271cc60dac2f8db29f25565d4be781b744292cdf6a861d53497d6: Status 404 returned error can't find the container with id f6a9ab90643271cc60dac2f8db29f25565d4be781b744292cdf6a861d53497d6 Nov 02 03:57:54.790870 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.790835 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" event={"ID":"9045f2f7-7e64-4a29-b7fa-d4c369063e5b","Type":"ContainerStarted","Data":"3f970f400185e3102eac510154c3a3736207217beedec3e8c69f1b0d8c3af35a"} Nov 02 03:57:54.791013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.790898 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" event={"ID":"9045f2f7-7e64-4a29-b7fa-d4c369063e5b","Type":"ContainerStarted","Data":"f6a9ab90643271cc60dac2f8db29f25565d4be781b744292cdf6a861d53497d6"} Nov 02 03:57:54.792233 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.792210 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" Nov 02 03:57:54.792376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.792250 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v" event={"ID":"70eea3d2-c0b0-44d4-a7b2-7078d918f9e8","Type":"ContainerDied","Data":"ae862ec3c119a8670f7375f8efac4ab2d402f965518f068e38cf8ba930ea8896"} Nov 02 03:57:54.792376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.792302 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae862ec3c119a8670f7375f8efac4ab2d402f965518f068e38cf8ba930ea8896" Nov 02 03:57:54.906496 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:54.906450 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" podStartSLOduration=1.906436907 podStartE2EDuration="1.906436907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:57:54.902860748 +0000 UTC m=+4216.898561691" watchObservedRunningTime="2026-11-02 03:57:54.906436907 +0000 UTC m=+4216.902137887" Nov 02 03:57:55.799743 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:55.799705 4868 generic.go:356] "Generic (PLEG): container finished" podID="9045f2f7-7e64-4a29-b7fa-d4c369063e5b" containerID="3f970f400185e3102eac510154c3a3736207217beedec3e8c69f1b0d8c3af35a" exitCode=0 Nov 02 03:57:55.799915 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:55.799786 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" event={"ID":"9045f2f7-7e64-4a29-b7fa-d4c369063e5b","Type":"ContainerDied","Data":"3f970f400185e3102eac510154c3a3736207217beedec3e8c69f1b0d8c3af35a"} Nov 02 03:57:56.959162 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:56.959129 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:57.086243 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.086164 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca\") pod \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " Nov 02 03:57:57.086243 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.086237 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc\" (UniqueName: \"kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc\") pod \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " Nov 02 03:57:57.086535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.086337 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca\") pod \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\" (UID: \"9045f2f7-7e64-4a29-b7fa-d4c369063e5b\") " Nov 02 03:57:57.086535 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.086519 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca" pod "9045f2f7-7e64-4a29-b7fa-d4c369063e5b" (UID: "9045f2f7-7e64-4a29-b7fa-d4c369063e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:57:57.086843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.086818 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:57.088366 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.088317 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca" pod "9045f2f7-7e64-4a29-b7fa-d4c369063e5b" (UID: "9045f2f7-7e64-4a29-b7fa-d4c369063e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:57:57.088590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.088558 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc" pod "9045f2f7-7e64-4a29-b7fa-d4c369063e5b" (UID: "9045f2f7-7e64-4a29-b7fa-d4c369063e5b"). InnerVolumeSpecName "kube-api-access-bv4cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:57:57.187658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.187625 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv4cc\" (UniqueName: \"kubernetes.io/projected/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-kube-api-access-bv4cc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:57.187658 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.187653 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9045f2f7-7e64-4a29-b7fa-d4c369063e5b-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:57:57.813547 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.813518 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" Nov 02 03:57:57.813725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.813552 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5" event={"ID":"9045f2f7-7e64-4a29-b7fa-d4c369063e5b","Type":"ContainerDied","Data":"f6a9ab90643271cc60dac2f8db29f25565d4be781b744292cdf6a861d53497d6"} Nov 02 03:57:57.813725 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:57:57.813602 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a9ab90643271cc60dac2f8db29f25565d4be781b744292cdf6a861d53497d6" Nov 02 03:58:00.901765 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.901730 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs"] Nov 02 03:58:00.902254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.902237 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9045f2f7-7e64-4a29-b7fa-d4c369063e5b" containerName="revoke-keys" Nov 02 03:58:00.902254 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.902254 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9045f2f7-7e64-4a29-b7fa-d4c369063e5b" containerName="revoke-keys" Nov 02 03:58:00.902414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.902272 4868 cpu_manager.go:414] "containerMap: removing container" podUID="70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" containerName="revoke-keys" Nov 02 03:58:00.902414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.902277 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" containerName="revoke-keys" Nov 02 03:58:00.902414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.902393 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9045f2f7-7e64-4a29-b7fa-d4c369063e5b" containerName="revoke-keys" Nov 02 03:58:00.902414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.902404 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" containerName="revoke-keys" Nov 02 03:58:00.905934 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.905914 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:00.961240 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:00.961212 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs"] Nov 02 03:58:01.007199 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.007164 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 03:58:01.007339 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.007205 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 03:58:01.028924 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.028882 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.029052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.028933 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7sjk\" (UniqueName: \"kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.029052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.028996 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.130621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.130577 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.130621 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.130626 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7sjk\" (UniqueName: \"kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.130843 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.130651 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.131215 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.131193 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.133003 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.132985 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.189949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.189880 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7sjk\" (UniqueName: \"kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk\") pod \"maas-api-revoke-sub-073cbf6d-vn6bs\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.219470 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.219438 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:01.421411 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.421373 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn"] Nov 02 03:58:01.426521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.426487 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.515483 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.510746 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ae1315-openshift-default-dockercfg-gtk98\"" Nov 02 03:58:01.526380 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.522505 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn"] Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536065 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536128 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536170 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536205 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536257 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536347 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536404 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536438 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshzc\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.537372 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.536469 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.643379 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.643315 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.643611 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.643596 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.643704 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.643692 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshzc\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.643774 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.643765 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.643905 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.643897 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.643981 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.643972 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.644052 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.644043 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.644120 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.644111 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.644201 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.644190 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.645046 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.645019 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.647266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.646252 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.647266 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.647211 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.647686 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.647666 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.648196 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.648154 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.650040 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.649995 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.650507 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.650490 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.676768 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.676746 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs"] Nov 02 03:58:01.678402 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:58:01.678336 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e694303_3925_4b9a_9fec_5e6aa7baaa91.slice/crio-768cab20e22b426d9db9adcbafedd64ad5100e81a1cb7f1451aceee290ccb759 WatchSource:0}: Error finding container 768cab20e22b426d9db9adcbafedd64ad5100e81a1cb7f1451aceee290ccb759: Status 404 returned error can't find the container with id 768cab20e22b426d9db9adcbafedd64ad5100e81a1cb7f1451aceee290ccb759 Nov 02 03:58:01.707173 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.705177 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.713023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.712994 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshzc\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc\") pod \"e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.741462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.741434 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:01.845026 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:01.844989 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" event={"ID":"0e694303-3925-4b9a-9fec-5e6aa7baaa91","Type":"ContainerStarted","Data":"768cab20e22b426d9db9adcbafedd64ad5100e81a1cb7f1451aceee290ccb759"} Nov 02 03:58:02.166927 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.166896 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn"] Nov 02 03:58:02.169105 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:58:02.169079 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24607332_f862_4be8_9712_8e1e7f6c1b68.slice/crio-b86262d5af105e7db56edd1d592afdf3695179cd78457a137d50de83a18c83f2 WatchSource:0}: Error finding container b86262d5af105e7db56edd1d592afdf3695179cd78457a137d50de83a18c83f2: Status 404 returned error can't find the container with id b86262d5af105e7db56edd1d592afdf3695179cd78457a137d50de83a18c83f2 Nov 02 03:58:02.176396 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.176345 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:58:02.176489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.176435 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:58:02.176489 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.176465 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:58:02.853048 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.853018 4868 generic.go:356] "Generic (PLEG): container finished" podID="0e694303-3925-4b9a-9fec-5e6aa7baaa91" containerID="c2550d1808147b9e1ed9ad7c72c61de29f8422e2dcabe699023beb132ec442da" exitCode=0 Nov 02 03:58:02.853250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.853090 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" event={"ID":"0e694303-3925-4b9a-9fec-5e6aa7baaa91","Type":"ContainerDied","Data":"c2550d1808147b9e1ed9ad7c72c61de29f8422e2dcabe699023beb132ec442da"} Nov 02 03:58:02.855343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.855302 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" event={"ID":"24607332-f862-4be8-9712-8e1e7f6c1b68","Type":"ContainerStarted","Data":"3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068"} Nov 02 03:58:02.855486 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:02.855370 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" event={"ID":"24607332-f862-4be8-9712-8e1e7f6c1b68","Type":"ContainerStarted","Data":"b86262d5af105e7db56edd1d592afdf3695179cd78457a137d50de83a18c83f2"} Nov 02 03:58:03.084280 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:03.084224 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podStartSLOduration=2.084209177 podStartE2EDuration="2.084209177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:58:03.082624223 +0000 UTC m=+4225.078325168" watchObservedRunningTime="2026-11-02 03:58:03.084209177 +0000 UTC m=+4225.079910120" Nov 02 03:58:03.445826 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:58:03.445791 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:58:03.742593 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:03.742507 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:03.748709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:03.748680 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:03.862498 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:03.862468 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:03.863538 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:03.863514 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 03:58:04.079414 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.079387 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:04.173811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.173778 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca\") pod \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " Nov 02 03:58:04.174017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.173832 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk\" (UniqueName: \"kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk\") pod \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " Nov 02 03:58:04.174017 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.173969 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca\") pod \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\" (UID: \"0e694303-3925-4b9a-9fec-5e6aa7baaa91\") " Nov 02 03:58:04.174212 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.174189 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca" pod "0e694303-3925-4b9a-9fec-5e6aa7baaa91" (UID: "0e694303-3925-4b9a-9fec-5e6aa7baaa91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:58:04.174374 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.174341 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:04.175962 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.175939 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca" pod "0e694303-3925-4b9a-9fec-5e6aa7baaa91" (UID: "0e694303-3925-4b9a-9fec-5e6aa7baaa91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:58:04.176061 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.176046 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk" pod "0e694303-3925-4b9a-9fec-5e6aa7baaa91" (UID: "0e694303-3925-4b9a-9fec-5e6aa7baaa91"). InnerVolumeSpecName "kube-api-access-g7sjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:58:04.230622 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.230579 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt"] Nov 02 03:58:04.231132 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.231110 4868 cpu_manager.go:414] "containerMap: removing container" podUID="0e694303-3925-4b9a-9fec-5e6aa7baaa91" containerName="revoke-keys" Nov 02 03:58:04.231132 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.231129 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e694303-3925-4b9a-9fec-5e6aa7baaa91" containerName="revoke-keys" Nov 02 03:58:04.231343 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.231298 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e694303-3925-4b9a-9fec-5e6aa7baaa91" containerName="revoke-keys" Nov 02 03:58:04.234856 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.234834 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.275274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.275248 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7sjk\" (UniqueName: \"kubernetes.io/projected/0e694303-3925-4b9a-9fec-5e6aa7baaa91-kube-api-access-g7sjk\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:04.275274 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.275274 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e694303-3925-4b9a-9fec-5e6aa7baaa91-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:04.289811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.289785 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt"] Nov 02 03:58:04.376433 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.376400 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.376586 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.376573 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk46s\" (UniqueName: \"kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.376685 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.376672 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.477589 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.477558 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.478010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.477696 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk46s\" (UniqueName: \"kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.478010 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.477815 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.478250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.478229 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.480059 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.480036 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.534225 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.534200 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk46s\" (UniqueName: \"kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s\") pod \"maas-api-revoke-sub-22ffe1e6-sjkwt\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.549148 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.549129 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:04.869301 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.869263 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" Nov 02 03:58:04.869534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.869298 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs" event={"ID":"0e694303-3925-4b9a-9fec-5e6aa7baaa91","Type":"ContainerDied","Data":"768cab20e22b426d9db9adcbafedd64ad5100e81a1cb7f1451aceee290ccb759"} Nov 02 03:58:04.869534 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.869373 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="768cab20e22b426d9db9adcbafedd64ad5100e81a1cb7f1451aceee290ccb759" Nov 02 03:58:04.919944 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:04.919918 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt"] Nov 02 03:58:04.921874 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:58:04.921848 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeaf4ac9_f2bc_414d_a7d5_c7af035ca68a.slice/crio-b7119fa0d75798316f1494d34424290636ab4d6c1051c5050971d96f149cf5c8 WatchSource:0}: Error finding container b7119fa0d75798316f1494d34424290636ab4d6c1051c5050971d96f149cf5c8: Status 404 returned error can't find the container with id b7119fa0d75798316f1494d34424290636ab4d6c1051c5050971d96f149cf5c8 Nov 02 03:58:05.876795 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:05.876758 4868 generic.go:356] "Generic (PLEG): container finished" podID="beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" containerID="686ce55a205631a92e6ed6ba7a1d0a191ca7f361c752c4aefc6550a8e2431152" exitCode=0 Nov 02 03:58:05.877220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:05.876848 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" event={"ID":"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a","Type":"ContainerDied","Data":"686ce55a205631a92e6ed6ba7a1d0a191ca7f361c752c4aefc6550a8e2431152"} Nov 02 03:58:05.877220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:05.876917 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" event={"ID":"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a","Type":"ContainerStarted","Data":"b7119fa0d75798316f1494d34424290636ab4d6c1051c5050971d96f149cf5c8"} Nov 02 03:58:07.058167 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.058140 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:07.211335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.211256 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca\") pod \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " Nov 02 03:58:07.211335 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.211302 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s\" (UniqueName: \"kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s\") pod \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " Nov 02 03:58:07.211590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.211398 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca\") pod \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\" (UID: \"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a\") " Nov 02 03:58:07.211645 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.211605 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca" pod "beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" (UID: "beaf4ac9-f2bc-414d-a7d5-c7af035ca68a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 03:58:07.211959 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.211930 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:07.213631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.213610 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s" pod "beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" (UID: "beaf4ac9-f2bc-414d-a7d5-c7af035ca68a"). InnerVolumeSpecName "kube-api-access-rk46s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:58:07.213631 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.213612 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca" pod "beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" (UID: "beaf4ac9-f2bc-414d-a7d5-c7af035ca68a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:58:07.312814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.312786 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk46s\" (UniqueName: \"kubernetes.io/projected/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-kube-api-access-rk46s\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:07.312814 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.312813 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:07.890528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.890493 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" event={"ID":"beaf4ac9-f2bc-414d-a7d5-c7af035ca68a","Type":"ContainerDied","Data":"b7119fa0d75798316f1494d34424290636ab4d6c1051c5050971d96f149cf5c8"} Nov 02 03:58:07.890528 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.890518 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt" Nov 02 03:58:07.890718 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:07.890542 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7119fa0d75798316f1494d34424290636ab4d6c1051c5050971d96f149cf5c8" Nov 02 03:58:11.950392 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.950325 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw"] Nov 02 03:58:11.951125 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.951101 4868 cpu_manager.go:414] "containerMap: removing container" podUID="beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" containerName="revoke-keys" Nov 02 03:58:11.951251 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.951127 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" containerName="revoke-keys" Nov 02 03:58:11.951251 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.951244 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" containerName="revoke-keys" Nov 02 03:58:11.955880 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.955859 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:11.989890 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.989862 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:11.990037 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:11.990013 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cccs\" (UniqueName: \"kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs\") pod \"payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:12.008175 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.008144 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw"] Nov 02 03:58:12.053850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.053826 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ae1315-dockercfg-2dr5s\"" Nov 02 03:58:12.053850 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.053826 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ae1315\"" Nov 02 03:58:12.090668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.090640 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:12.090868 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.090847 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cccs\" (UniqueName: \"kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs\") pod \"payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:12.091605 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.091581 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:12.147178 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.147149 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cccs\" (UniqueName: \"kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs\") pod \"payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:12.274464 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.274391 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:12.649773 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.649745 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw"] Nov 02 03:58:12.650845 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:58:12.650814 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8f7864_786f_4f78_86ab_63769cb63896.slice/crio-69567e600e414ea590670b14857220ead4ea1b1ec1cfdecc85a5b2d1f8902742 WatchSource:0}: Error finding container 69567e600e414ea590670b14857220ead4ea1b1ec1cfdecc85a5b2d1f8902742: Status 404 returned error can't find the container with id 69567e600e414ea590670b14857220ead4ea1b1ec1cfdecc85a5b2d1f8902742 Nov 02 03:58:12.946318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.946235 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" event={"ID":"3c8f7864-786f-4f78-86ab-63769cb63896","Type":"ContainerStarted","Data":"317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356"} Nov 02 03:58:12.946318 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.946299 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" event={"ID":"3c8f7864-786f-4f78-86ab-63769cb63896","Type":"ContainerStarted","Data":"69567e600e414ea590670b14857220ead4ea1b1ec1cfdecc85a5b2d1f8902742"} Nov 02 03:58:12.946536 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:12.946378 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:13.061038 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:13.060975 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" podStartSLOduration=2.060955705 podStartE2EDuration="2.060955705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:58:13.057919066 +0000 UTC m=+4235.053620009" watchObservedRunningTime="2026-11-02 03:58:13.060955705 +0000 UTC m=+4235.056656648" Nov 02 03:58:23.957079 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:23.957046 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 03:58:28.294929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:28.294895 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz"] Nov 02 03:58:33.446075 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:58:33.446039 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:58:43.315235 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.315194 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" podUID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" containerName="main" containerID="cri-o://9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7" gracePeriod=45 Nov 02 03:58:43.338857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.338827 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" podUID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": dial tcp 10.132.0.169:8000: connect: connection refused" Nov 02 03:58:43.568537 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.568472 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:58:43.659117 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659089 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg\" (UniqueName: \"kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg\") pod \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " Nov 02 03:58:43.659287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659158 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs\") pod \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " Nov 02 03:58:43.659287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659183 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir\") pod \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " Nov 02 03:58:43.659287 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659219 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home\") pod \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " Nov 02 03:58:43.659491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659293 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm\") pod \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " Nov 02 03:58:43.659491 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659390 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache\") pod \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\" (UID: \"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898\") " Nov 02 03:58:43.659624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659492 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir" pod "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" (UID: "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:58:43.659624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659599 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home" pod "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" (UID: "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:58:43.659734 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659628 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache" pod "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" (UID: "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:58:43.659873 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659854 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:43.659941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659876 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:43.659941 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.659886 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:43.661479 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.661453 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm" pod "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" (UID: "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:58:43.661577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.661497 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg" pod "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" (UID: "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898"). InnerVolumeSpecName "kube-api-access-psbjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:58:43.661577 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.661512 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs" pod "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" (UID: "1c94f5ac-d6b1-4e4a-ac96-5e88c6530898"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:58:43.760971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.760942 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psbjg\" (UniqueName: \"kubernetes.io/projected/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-kube-api-access-psbjg\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:43.760971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.760969 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:43.760971 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:43.760979 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:58:44.135983 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.135954 4868 generic.go:356] "Generic (PLEG): container finished" podID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" containerID="9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7" exitCode=0 Nov 02 03:58:44.136124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.135996 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" event={"ID":"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898","Type":"ContainerDied","Data":"9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7"} Nov 02 03:58:44.136124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.136000 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" Nov 02 03:58:44.136124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.136029 4868 scope.go:122] "RemoveContainer" containerID="9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7" Nov 02 03:58:44.136124 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.136020 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz" event={"ID":"1c94f5ac-d6b1-4e4a-ac96-5e88c6530898","Type":"ContainerDied","Data":"c6d617fc27fde381ece623f17478ad79204eee5aa2c2cb4a286870a43d291341"} Nov 02 03:58:44.145025 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.145005 4868 scope.go:122] "RemoveContainer" containerID="9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7" Nov 02 03:58:44.145283 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:58:44.145264 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7\": container with ID starting with 9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7 not found: ID does not exist" containerID="9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7" Nov 02 03:58:44.145337 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.145291 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7"} err="failed to get container status \"9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7\": rpc error: code = NotFound desc = could not find container \"9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7\": container with ID starting with 9d49272ce48570e8134901e8a675926f3e18acbb203c9845a69a526c49a61cb7 not found: ID does not exist" Nov 02 03:58:44.298569 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.298538 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz"] Nov 02 03:58:44.347961 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:44.347930 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7c1bee/auth-test-model-7c1bee-kserve-9854985c7-xt9fz"] Nov 02 03:58:46.134587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.134553 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" path="/var/lib/kubelet/pods/1c94f5ac-d6b1-4e4a-ac96-5e88c6530898/volumes" Nov 02 03:58:46.205023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.204989 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk"] Nov 02 03:58:46.205527 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.205513 4868 cpu_manager.go:414] "containerMap: removing container" podUID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" containerName="main" Nov 02 03:58:46.205579 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.205528 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" containerName="main" Nov 02 03:58:46.205622 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.205613 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c94f5ac-d6b1-4e4a-ac96-5e88c6530898" containerName="main" Nov 02 03:58:46.209877 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.209861 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.262752 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.262725 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-61f3b6\"" Nov 02 03:58:46.268633 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.268606 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk"] Nov 02 03:58:46.318024 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.317992 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-61f3b6-dockercfg-bv8qh\"" Nov 02 03:58:46.319624 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.318881 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvrd\" (UniqueName: \"kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd\") pod \"payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.320059 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.320006 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.421857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.421763 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.422033 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.421979 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvrd\" (UniqueName: \"kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd\") pod \"payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.422590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.422566 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.478958 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.478927 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvrd\" (UniqueName: \"kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd\") pod \"payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.524590 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.524558 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:46.897256 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:46.897229 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk"] Nov 02 03:58:46.898691 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:58:46.898662 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae2a81c_f24b_46c9_88c3_0721ef2954d6.slice/crio-5547a199da62f3a1e2b8afa4f192270e40cb172b4bddf70f9bfe0480addf10b0 WatchSource:0}: Error finding container 5547a199da62f3a1e2b8afa4f192270e40cb172b4bddf70f9bfe0480addf10b0: Status 404 returned error can't find the container with id 5547a199da62f3a1e2b8afa4f192270e40cb172b4bddf70f9bfe0480addf10b0 Nov 02 03:58:47.156955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:47.156871 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" event={"ID":"9ae2a81c-f24b-46c9-88c3-0721ef2954d6","Type":"ContainerStarted","Data":"3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f"} Nov 02 03:58:47.156955 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:47.156925 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" event={"ID":"9ae2a81c-f24b-46c9-88c3-0721ef2954d6","Type":"ContainerStarted","Data":"5547a199da62f3a1e2b8afa4f192270e40cb172b4bddf70f9bfe0480addf10b0"} Nov 02 03:58:47.157376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:47.157055 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:58:47.269131 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:47.269067 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" podStartSLOduration=1.26905043 podStartE2EDuration="1.26905043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:58:47.268240734 +0000 UTC m=+4269.263941701" watchObservedRunningTime="2026-11-02 03:58:47.26905043 +0000 UTC m=+4269.264751375" Nov 02 03:58:53.519185 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:53.519085 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6"] Nov 02 03:58:58.181643 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:58:58.181614 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 03:59:03.446800 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:59:03.446763 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:59:08.544972 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.544922 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" podUID="d79d205c-ad61-4efe-a221-cb59738dedb0" containerName="main" containerID="cri-o://5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30" gracePeriod=45 Nov 02 03:59:08.816837 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.816805 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:59:08.896942 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.896915 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs\") pod \"d79d205c-ad61-4efe-a221-cb59738dedb0\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " Nov 02 03:59:08.897109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.896967 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh\" (UniqueName: \"kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh\") pod \"d79d205c-ad61-4efe-a221-cb59738dedb0\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " Nov 02 03:59:08.897109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897007 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm\") pod \"d79d205c-ad61-4efe-a221-cb59738dedb0\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " Nov 02 03:59:08.897109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897050 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir\") pod \"d79d205c-ad61-4efe-a221-cb59738dedb0\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " Nov 02 03:59:08.897109 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897092 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache\") pod \"d79d205c-ad61-4efe-a221-cb59738dedb0\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " Nov 02 03:59:08.897388 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897154 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home\") pod \"d79d205c-ad61-4efe-a221-cb59738dedb0\" (UID: \"d79d205c-ad61-4efe-a221-cb59738dedb0\") " Nov 02 03:59:08.897474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897415 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir" pod "d79d205c-ad61-4efe-a221-cb59738dedb0" (UID: "d79d205c-ad61-4efe-a221-cb59738dedb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:59:08.897474 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897461 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache" pod "d79d205c-ad61-4efe-a221-cb59738dedb0" (UID: "d79d205c-ad61-4efe-a221-cb59738dedb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:59:08.897596 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897540 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home" pod "d79d205c-ad61-4efe-a221-cb59738dedb0" (UID: "d79d205c-ad61-4efe-a221-cb59738dedb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:59:08.897742 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897715 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:59:08.897811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897746 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:59:08.897811 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.897762 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:59:08.899250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.899228 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm" pod "d79d205c-ad61-4efe-a221-cb59738dedb0" (UID: "d79d205c-ad61-4efe-a221-cb59738dedb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 03:59:08.899384 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.899363 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh" pod "d79d205c-ad61-4efe-a221-cb59738dedb0" (UID: "d79d205c-ad61-4efe-a221-cb59738dedb0"). InnerVolumeSpecName "kube-api-access-fp2gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 03:59:08.899437 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.899387 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs" pod "d79d205c-ad61-4efe-a221-cb59738dedb0" (UID: "d79d205c-ad61-4efe-a221-cb59738dedb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 03:59:08.999312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.999284 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d79d205c-ad61-4efe-a221-cb59738dedb0-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:59:08.999312 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.999313 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2gh\" (UniqueName: \"kubernetes.io/projected/d79d205c-ad61-4efe-a221-cb59738dedb0-kube-api-access-fp2gh\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:59:08.999525 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:08.999323 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d79d205c-ad61-4efe-a221-cb59738dedb0-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 03:59:09.308804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.308770 4868 generic.go:356] "Generic (PLEG): container finished" podID="d79d205c-ad61-4efe-a221-cb59738dedb0" containerID="5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30" exitCode=0 Nov 02 03:59:09.308979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.308813 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" Nov 02 03:59:09.308979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.308810 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" event={"ID":"d79d205c-ad61-4efe-a221-cb59738dedb0","Type":"ContainerDied","Data":"5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30"} Nov 02 03:59:09.308979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.308926 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6" event={"ID":"d79d205c-ad61-4efe-a221-cb59738dedb0","Type":"ContainerDied","Data":"6046b27cf60b255ccb131fca3bab30ef1442f254ac67577c12db323149efea70"} Nov 02 03:59:09.308979 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.308943 4868 scope.go:122] "RemoveContainer" containerID="5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30" Nov 02 03:59:09.319826 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.319804 4868 scope.go:122] "RemoveContainer" containerID="5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30" Nov 02 03:59:09.320079 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:59:09.320058 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30\": container with ID starting with 5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30 not found: ID does not exist" containerID="5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30" Nov 02 03:59:09.320137 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.320088 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30"} err="failed to get container status \"5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30\": rpc error: code = NotFound desc = could not find container \"5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30\": container with ID starting with 5751a484b914e7b1cf9fa825e253503e0b2d1057c8967f6fd7674a037f8ffb30 not found: ID does not exist" Nov 02 03:59:09.472709 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.472676 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6"] Nov 02 03:59:09.522262 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:09.522230 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aad578/auth-test-model-aad578-kserve-bf555c7d9-kr8b6"] Nov 02 03:59:10.115780 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:10.115747 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79d205c-ad61-4efe-a221-cb59738dedb0" path="/var/lib/kubelet/pods/d79d205c-ad61-4efe-a221-cb59738dedb0/volumes" Nov 02 03:59:18.412560 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.412521 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7"] Nov 02 03:59:18.413250 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.413232 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d79d205c-ad61-4efe-a221-cb59738dedb0" containerName="main" Nov 02 03:59:18.413300 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.413251 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79d205c-ad61-4efe-a221-cb59738dedb0" containerName="main" Nov 02 03:59:18.413381 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.413365 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79d205c-ad61-4efe-a221-cb59738dedb0" containerName="main" Nov 02 03:59:18.416760 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.416741 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.469220 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.469188 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-e69cf2-openshift-default-dockercfg-874t9\"" Nov 02 03:59:18.514030 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.513998 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514370 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514320 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514538 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514427 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514538 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514491 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514611 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514668 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514647 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsl6f\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514671 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514782 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514748 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.514891 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.514784 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.522759 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.522732 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7"] Nov 02 03:59:18.616023 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.615987 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616037 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616123 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616145 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616213 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616180 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616221 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616305 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616338 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsl6f\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616472 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616386 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616592 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616786 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616686 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.616894 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616805 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.617015 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.616980 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.617154 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.617037 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.618798 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.618776 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.618968 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.618952 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.691422 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.687566 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.691422 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.687932 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsl6f\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f\") pod \"e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:18.731929 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:18.730626 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:19.198308 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.198282 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7"] Nov 02 03:59:19.199174 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:59:19.199139 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01774e87_b148_4801_b4a8_7bf0564cd846.slice/crio-a4972e9798cedf0fd81598e56c9add53671b0d8c127f41d813cac3af1153c5c5 WatchSource:0}: Error finding container a4972e9798cedf0fd81598e56c9add53671b0d8c127f41d813cac3af1153c5c5: Status 404 returned error can't find the container with id a4972e9798cedf0fd81598e56c9add53671b0d8c127f41d813cac3af1153c5c5 Nov 02 03:59:19.201974 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.201938 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:59:19.202067 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.202011 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:59:19.202067 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.202042 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:59:19.377462 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.377428 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" event={"ID":"01774e87-b148-4801-b4a8-7bf0564cd846","Type":"ContainerStarted","Data":"bb60c3eb662b9c8789a7f70d10296c68079ca715b87193c358919c464cf6fc29"} Nov 02 03:59:19.377648 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.377482 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" event={"ID":"01774e87-b148-4801-b4a8-7bf0564cd846","Type":"ContainerStarted","Data":"a4972e9798cedf0fd81598e56c9add53671b0d8c127f41d813cac3af1153c5c5"} Nov 02 03:59:19.496439 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.496312 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podStartSLOduration=1.496292207 podStartE2EDuration="1.496292207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:59:19.493386932 +0000 UTC m=+4301.489087875" watchObservedRunningTime="2026-11-02 03:59:19.496292207 +0000 UTC m=+4301.491993152" Nov 02 03:59:19.731636 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:19.731600 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:20.737580 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:20.737554 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:21.390289 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:21.390253 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:21.391420 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:21.391396 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 03:59:26.236966 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.236931 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2"] Nov 02 03:59:26.241013 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.240993 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.297432 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.297396 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2"] Nov 02 03:59:26.359732 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.359699 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-e69cf2\"" Nov 02 03:59:26.359925 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.359812 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-e69cf2-dockercfg-z4v4s\"" Nov 02 03:59:26.398608 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.398570 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgz8\" (UniqueName: \"kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8\") pod \"payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.398949 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.398875 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.405736 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.405703 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw"] Nov 02 03:59:26.412904 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.412884 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.467587 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.467559 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw"] Nov 02 03:59:26.499568 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.499492 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.499728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.499564 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fgx8\" (UniqueName: \"kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8\") pod \"payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.499728 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.499612 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume\") pod \"payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.499866 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.499776 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgz8\" (UniqueName: \"kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8\") pod \"payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.500295 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.500274 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.556600 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.556565 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgz8\" (UniqueName: \"kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8\") pod \"payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:26.601157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.601130 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fgx8\" (UniqueName: \"kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8\") pod \"payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.601303 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.601175 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume\") pod \"payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.601853 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.601834 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume\") pod \"payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.657521 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.657490 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fgx8\" (UniqueName: \"kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8\") pod \"payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.734103 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.734068 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:26.854836 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:26.854798 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:27.109082 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.109057 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw"] Nov 02 03:59:27.110143 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:59:27.110110 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf645cb_7832_4037_964e_512b37c92926.slice/crio-cbad238ba9d7a4d1f7aef7c3cea1af28a90c79beaabe02a410322337a3c60b47 WatchSource:0}: Error finding container cbad238ba9d7a4d1f7aef7c3cea1af28a90c79beaabe02a410322337a3c60b47: Status 404 returned error can't find the container with id cbad238ba9d7a4d1f7aef7c3cea1af28a90c79beaabe02a410322337a3c60b47 Nov 02 03:59:27.232183 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.232156 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2"] Nov 02 03:59:27.233209 ip-10-0-137-74 kubenswrapper[4868]: W1102 03:59:27.233183 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8274f7b0_d23f_4200_870f_252676b95f5f.slice/crio-db051ce0fe9984c3b6a58fc06f8999f5019882c9c76bb94b6df40eed9c3be3b1 WatchSource:0}: Error finding container db051ce0fe9984c3b6a58fc06f8999f5019882c9c76bb94b6df40eed9c3be3b1: Status 404 returned error can't find the container with id db051ce0fe9984c3b6a58fc06f8999f5019882c9c76bb94b6df40eed9c3be3b1 Nov 02 03:59:27.436407 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.436270 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" event={"ID":"ccf645cb-7832-4037-964e-512b37c92926","Type":"ContainerStarted","Data":"0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d"} Nov 02 03:59:27.436858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.436508 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:27.436858 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.436556 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" event={"ID":"ccf645cb-7832-4037-964e-512b37c92926","Type":"ContainerStarted","Data":"cbad238ba9d7a4d1f7aef7c3cea1af28a90c79beaabe02a410322337a3c60b47"} Nov 02 03:59:27.438028 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.438004 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" event={"ID":"8274f7b0-d23f-4200-870f-252676b95f5f","Type":"ContainerStarted","Data":"f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267"} Nov 02 03:59:27.438157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.438046 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" event={"ID":"8274f7b0-d23f-4200-870f-252676b95f5f","Type":"ContainerStarted","Data":"db051ce0fe9984c3b6a58fc06f8999f5019882c9c76bb94b6df40eed9c3be3b1"} Nov 02 03:59:27.438157 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.438091 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:27.562682 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.562620 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" podStartSLOduration=1.562600422 podStartE2EDuration="1.562600422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:59:27.561589402 +0000 UTC m=+4309.557290347" watchObservedRunningTime="2026-11-02 03:59:27.562600422 +0000 UTC m=+4309.558301366" Nov 02 03:59:27.678669 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:27.678617 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" podStartSLOduration=1.6786027209999999 podStartE2EDuration="1.678602721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 03:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 03:59:27.675158161 +0000 UTC m=+4309.670859104" watchObservedRunningTime="2026-11-02 03:59:27.678602721 +0000 UTC m=+4309.674303663" Nov 02 03:59:33.445768 ip-10-0-137-74 kubenswrapper[4868]: E1102 03:59:33.445730 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 03:59:38.457327 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:38.457290 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 03:59:38.458650 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:38.458627 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 03:59:42.639497 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:42.639445 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 03:59:42.640043 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:42.639517 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 03:59:45.563278 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:45.563237 4868 generic.go:356] "Generic (PLEG): container finished" podID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerID="bc0bbc40c7a039e929b15414f149ce93775b64368895634f041d1ef42d7637e0" exitCode=0 Nov 02 03:59:45.563804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:45.563311 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" event={"ID":"80ae4893-94e4-484c-bac3-c1f582c7d863","Type":"ContainerDied","Data":"bc0bbc40c7a039e929b15414f149ce93775b64368895634f041d1ef42d7637e0"} Nov 02 03:59:45.563804 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:45.563764 4868 scope.go:122] "RemoveContainer" containerID="bc0bbc40c7a039e929b15414f149ce93775b64368895634f041d1ef42d7637e0" Nov 02 03:59:45.565448 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:45.565422 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:59:45.565529 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:45.565492 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:59:45.565572 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:45.565535 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 03:59:46.571538 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:46.571504 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" event={"ID":"80ae4893-94e4-484c-bac3-c1f582c7d863","Type":"ContainerStarted","Data":"563cdb1a4a9b9689f5062de8412c305ec5374d0b5238ab9db73d45d90c548d29"} Nov 02 03:59:46.847857 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:46.847763 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:59:46.848242 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:46.848156 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 03:59:46.848442 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:46.848419 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 03:59:47.849376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:47.848730 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 03:59:47.849376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:47.848815 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 03:59:48.842800 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:48.842756 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 03:59:48.843119 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:48.843083 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 03:59:49.849385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:49.848741 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 03:59:49.849385 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:49.848822 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 03:59:50.853376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:50.852717 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body= Nov 02 03:59:50.853376 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:50.852799 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" Nov 02 03:59:51.837526 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:51.837490 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:59:51.837948 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:51.837926 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 03:59:51.838642 ip-10-0-137-74 kubenswrapper[4868]: I1102 03:59:51.838624 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 04:00:00.165918 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.165883 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c"] Nov 02 04:00:00.170448 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.170422 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp"] Nov 02 04:00:00.170603 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.170556 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.174103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.174081 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds"] Nov 02 04:00:00.174233 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.174196 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.177691 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.177674 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.215594 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.215568 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q"] Nov 02 04:00:00.220093 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.220071 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf"] Nov 02 04:00:00.220214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.220164 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.224564 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.224543 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp"] Nov 02 04:00:00.224709 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.224689 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.230263 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.230244 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6"] Nov 02 04:00:00.230401 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.230384 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.254251 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.254220 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4"] Nov 02 04:00:00.254426 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.254408 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.260025 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.260004 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv"] Nov 02 04:00:00.260141 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.260119 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.262407 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.262347 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:00:00.264434 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.264270 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n"] Nov 02 04:00:00.264434 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.264391 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.268262 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.268243 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz"] Nov 02 04:00:00.268388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.268341 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.272472 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.272455 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp"] Nov 02 04:00:00.272586 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.272483 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c"] Nov 02 04:00:00.272586 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.272546 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.274258 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.274236 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds"] Nov 02 04:00:00.280027 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280004 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.280121 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280039 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppxv\" (UniqueName: \"kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.280167 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280130 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmlc\" (UniqueName: \"kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc\") pod \"maas-api-key-cleanup-29893200-4lxjf\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.280204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280165 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnnqm\" (UniqueName: \"kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.280204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280191 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893200-4lxjf\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.280286 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280268 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd77\" (UniqueName: \"kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.280329 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280307 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.280383 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280327 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7znw\" (UniqueName: \"kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.280431 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280417 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t56c\" (UniqueName: \"kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.280474 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280444 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.280474 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280461 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.280557 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.280538 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.303957 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.303935 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf"] Nov 02 04:00:00.307966 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.307940 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q"] Nov 02 04:00:00.330573 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.317580 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6"] Nov 02 04:00:00.373033 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.372146 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv"] Nov 02 04:00:00.376935 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.376907 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n"] Nov 02 04:00:00.380419 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.380396 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz"] Nov 02 04:00:00.382097 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382067 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnnqm\" (UniqueName: \"kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.382218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382121 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.382218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382153 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.382218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382178 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.382218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382208 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893200-4lxjf\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.382525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382503 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd77\" (UniqueName: \"kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.382621 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382585 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.382698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382615 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7znw\" (UniqueName: \"kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.382766 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382711 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t56c\" (UniqueName: \"kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.382766 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382747 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.382890 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382767 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.382890 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382808 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsld2\" (UniqueName: \"kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.383040 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382893 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.383040 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382933 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.383040 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.382988 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdh6\" (UniqueName: \"kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.383225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383047 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.383225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383054 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893200-4lxjf\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383485 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383625 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383721 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383741 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppxv\" (UniqueName: \"kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383760 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383786 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktqk\" (UniqueName: \"kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383810 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2l4\" (UniqueName: \"kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383841 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmlc\" (UniqueName: \"kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc\") pod \"maas-api-key-cleanup-29893200-4lxjf\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.383952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.383855 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.384646 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.384284 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.384646 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.384535 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.396297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.396253 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4"] Nov 02 04:00:00.399851 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.399828 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp"] Nov 02 04:00:00.440341 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.440269 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnnqm\" (UniqueName: \"kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.440341 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.440295 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd77\" (UniqueName: \"kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.440726 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.440696 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmlc\" (UniqueName: \"kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc\") pod \"maas-api-key-cleanup-29893200-4lxjf\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.440850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.440763 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppxv\" (UniqueName: \"kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv\") pod \"maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.442060 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.442036 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7znw\" (UniqueName: \"kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.442204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.442184 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t56c\" (UniqueName: \"kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.484627 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.484600 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:00.484992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.484972 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdh6\" (UniqueName: \"kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.485061 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485012 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.485163 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485140 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.485223 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485191 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktqk\" (UniqueName: \"kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.485283 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485228 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2l4\" (UniqueName: \"kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.485343 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485302 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.485343 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485331 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.485481 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485347 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.485541 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485485 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsld2\" (UniqueName: \"kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.485541 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485525 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.485796 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.485769 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.486046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.486029 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.486235 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.486216 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.486272 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.486246 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.486321 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.486306 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.493138 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.493119 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:00.500195 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.500175 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:00.542704 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.542647 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdh6\" (UniqueName: \"kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.543236 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.543211 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsld2\" (UniqueName: \"kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.543396 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.543335 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktqk\" (UniqueName: \"kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk\") pod \"maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.543454 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.543385 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2l4\" (UniqueName: \"kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.543975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.543954 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.561392 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.559049 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:00.565306 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.564678 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:00.574913 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.572875 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:00.580585 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.580563 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:00.589693 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.589517 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:00.596017 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.595833 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:00.609964 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.609807 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:00.617561 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.617045 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:00.876533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:00.876503 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c"] Nov 02 04:00:00.877232 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:00.877207 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e19ea6_be5d_459a_a313_cb6b8d925fd3.slice/crio-a2953f9d00da532211740b15285dd58c9c065d6e1741a5cea5ef14341fa6db92 WatchSource:0}: Error finding container a2953f9d00da532211740b15285dd58c9c065d6e1741a5cea5ef14341fa6db92: Status 404 returned error can't find the container with id a2953f9d00da532211740b15285dd58c9c065d6e1741a5cea5ef14341fa6db92 Nov 02 04:00:01.092709 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.092676 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11967c4f_0f40_4152_8e76_2f5648550693.slice/crio-e60c42249923a33b33da12b295966fec7191f90840d2b6ad7affe3d6073eb31f WatchSource:0}: Error finding container e60c42249923a33b33da12b295966fec7191f90840d2b6ad7affe3d6073eb31f: Status 404 returned error can't find the container with id e60c42249923a33b33da12b295966fec7191f90840d2b6ad7affe3d6073eb31f Nov 02 04:00:01.093471 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.093448 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds"] Nov 02 04:00:01.116247 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.116224 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp"] Nov 02 04:00:01.125511 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.125491 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7258881_8b2d_4a65_8dc6_e59d2d13701c.slice/crio-4690786171d640f65cb6677960dfe1fa4cd70babd37adec21cdb536d098c9fea WatchSource:0}: Error finding container 4690786171d640f65cb6677960dfe1fa4cd70babd37adec21cdb536d098c9fea: Status 404 returned error can't find the container with id 4690786171d640f65cb6677960dfe1fa4cd70babd37adec21cdb536d098c9fea Nov 02 04:00:01.140764 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.140734 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fe7ef7_fb83_47d1_89db_1b9d898bae39.slice/crio-17fc43493c71613c07305943758bd9b6855f69b2e28cbcc069d48b84696c1530 WatchSource:0}: Error finding container 17fc43493c71613c07305943758bd9b6855f69b2e28cbcc069d48b84696c1530: Status 404 returned error can't find the container with id 17fc43493c71613c07305943758bd9b6855f69b2e28cbcc069d48b84696c1530 Nov 02 04:00:01.141675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.141605 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6"] Nov 02 04:00:01.141840 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.141816 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294241de_b2e0_4c92_af80_dfdbaa6060c3.slice/crio-099b57705721b877cce063e622b3771b580ca568c879d36d3823cf831c1360b6 WatchSource:0}: Error finding container 099b57705721b877cce063e622b3771b580ca568c879d36d3823cf831c1360b6: Status 404 returned error can't find the container with id 099b57705721b877cce063e622b3771b580ca568c879d36d3823cf831c1360b6 Nov 02 04:00:01.146211 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.146176 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz"] Nov 02 04:00:01.207371 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.207327 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92b6429_925b_420d_b916_fdf9f30e979c.slice/crio-bb0fa68e02a3dc973f2608abd9eff39ec9b56ace603f6ac7394fe773a1805780 WatchSource:0}: Error finding container bb0fa68e02a3dc973f2608abd9eff39ec9b56ace603f6ac7394fe773a1805780: Status 404 returned error can't find the container with id bb0fa68e02a3dc973f2608abd9eff39ec9b56ace603f6ac7394fe773a1805780 Nov 02 04:00:01.207648 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.207570 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q"] Nov 02 04:00:01.209479 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.209456 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09169936_e489_4c22_9d56_8478c2436f68.slice/crio-e025c05abe9fed748dfbd5e7d8ca0b4a70a0c9073ac28ecc3b7ed295a06f8aed WatchSource:0}: Error finding container e025c05abe9fed748dfbd5e7d8ca0b4a70a0c9073ac28ecc3b7ed295a06f8aed: Status 404 returned error can't find the container with id e025c05abe9fed748dfbd5e7d8ca0b4a70a0c9073ac28ecc3b7ed295a06f8aed Nov 02 04:00:01.211138 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.211118 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp"] Nov 02 04:00:01.220862 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.220843 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf"] Nov 02 04:00:01.225431 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.225244 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv"] Nov 02 04:00:01.231448 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.231422 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3760203b_bcac_4f6a_a50b_9943eedb32aa.slice/crio-8d888ed83435fdf59703dca6ff1287e0a3073e5d1d673f7536a59ff9d7ae120c WatchSource:0}: Error finding container 8d888ed83435fdf59703dca6ff1287e0a3073e5d1d673f7536a59ff9d7ae120c: Status 404 returned error can't find the container with id 8d888ed83435fdf59703dca6ff1287e0a3073e5d1d673f7536a59ff9d7ae120c Nov 02 04:00:01.265957 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.265924 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85e0a2b_8ce6_4b15_bb89_d017abfb072d.slice/crio-2367090f809033cb682d6272e3fd54ed3ce173e64aac5635c20d18576c0934b7 WatchSource:0}: Error finding container 2367090f809033cb682d6272e3fd54ed3ce173e64aac5635c20d18576c0934b7: Status 404 returned error can't find the container with id 2367090f809033cb682d6272e3fd54ed3ce173e64aac5635c20d18576c0934b7 Nov 02 04:00:01.266144 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.266117 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n"] Nov 02 04:00:01.302974 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.302951 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4"] Nov 02 04:00:01.303897 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:01.303874 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade750b7_8be1_45da_aeb2_cfd2e6cef52b.slice/crio-9c63cdd640948271b8b313080c823456b7746c84a38bb2559958ee0ff134589e WatchSource:0}: Error finding container 9c63cdd640948271b8b313080c823456b7746c84a38bb2559958ee0ff134589e: Status 404 returned error can't find the container with id 9c63cdd640948271b8b313080c823456b7746c84a38bb2559958ee0ff134589e Nov 02 04:00:01.715801 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.715761 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerStarted","Data":"2367090f809033cb682d6272e3fd54ed3ce173e64aac5635c20d18576c0934b7"} Nov 02 04:00:01.717141 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.717113 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerStarted","Data":"9c63cdd640948271b8b313080c823456b7746c84a38bb2559958ee0ff134589e"} Nov 02 04:00:01.718606 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.718580 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerStarted","Data":"06dc054da5fe4e688246d54909acfc8a37089a63c0b6477093976177ed6b4477"} Nov 02 04:00:01.718700 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.718630 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerStarted","Data":"a2953f9d00da532211740b15285dd58c9c065d6e1741a5cea5ef14341fa6db92"} Nov 02 04:00:01.720279 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.720254 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerStarted","Data":"e018205305bcb561af1b393ae4f54e293a54b41a2311d98379a3d55eb634455d"} Nov 02 04:00:01.720424 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.720304 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerStarted","Data":"e60c42249923a33b33da12b295966fec7191f90840d2b6ad7affe3d6073eb31f"} Nov 02 04:00:01.721812 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.721790 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerStarted","Data":"0900f13caebdc67138a5a697c6af03f93138a7dae929ff7c34b9f7d5613d6987"} Nov 02 04:00:01.721896 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.721827 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerStarted","Data":"099b57705721b877cce063e622b3771b580ca568c879d36d3823cf831c1360b6"} Nov 02 04:00:01.723047 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.723026 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerStarted","Data":"264e8a3f350c7c9d54fa69cc6ce68e901fdbc609b65bc86bd3a65fdeaa7a71b0"} Nov 02 04:00:01.724120 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.724101 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerStarted","Data":"e025c05abe9fed748dfbd5e7d8ca0b4a70a0c9073ac28ecc3b7ed295a06f8aed"} Nov 02 04:00:01.725478 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.725454 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerStarted","Data":"b1835aa521fe2d085e6505de27bcaa011401b4a1c0c20bd71a24676609ff4f0d"} Nov 02 04:00:01.725560 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.725500 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerStarted","Data":"bb0fa68e02a3dc973f2608abd9eff39ec9b56ace603f6ac7394fe773a1805780"} Nov 02 04:00:01.727010 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.726988 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerStarted","Data":"f4bafb9baf43922f250d9fec03144293b8d84a927999d19ac81c9c5ecf482437"} Nov 02 04:00:01.727113 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.727036 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerStarted","Data":"17fc43493c71613c07305943758bd9b6855f69b2e28cbcc069d48b84696c1530"} Nov 02 04:00:01.728552 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.728531 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerStarted","Data":"5e88d5675af913db530d942d8f08eee76107c0da9ba4ed14eb6b2415ac09d771"} Nov 02 04:00:01.728628 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.728570 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerStarted","Data":"4690786171d640f65cb6677960dfe1fa4cd70babd37adec21cdb536d098c9fea"} Nov 02 04:00:01.730111 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.730065 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerStarted","Data":"913af3404be10b51596f51933133b66bee3709f73452f94e187fcc02f99120d6"} Nov 02 04:00:01.730207 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.730131 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerStarted","Data":"8d888ed83435fdf59703dca6ff1287e0a3073e5d1d673f7536a59ff9d7ae120c"} Nov 02 04:00:01.830425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.830227 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" podStartSLOduration=1.83021107 podStartE2EDuration="1.83021107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:01.829898981 +0000 UTC m=+4343.825599926" watchObservedRunningTime="2026-11-02 04:00:01.83021107 +0000 UTC m=+4343.825912013" Nov 02 04:00:01.945437 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:01.945316 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" podStartSLOduration=1.945301916 podStartE2EDuration="1.945301916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:01.94267797 +0000 UTC m=+4343.938378915" watchObservedRunningTime="2026-11-02 04:00:01.945301916 +0000 UTC m=+4343.941002853" Nov 02 04:00:02.056798 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.056736 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" podStartSLOduration=2.05672125 podStartE2EDuration="2.05672125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:02.054049161 +0000 UTC m=+4344.049750106" watchObservedRunningTime="2026-11-02 04:00:02.05672125 +0000 UTC m=+4344.052422240" Nov 02 04:00:02.170295 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.170243 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" podStartSLOduration=2.170226525 podStartE2EDuration="2.170226525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:02.167701725 +0000 UTC m=+4344.163402668" watchObservedRunningTime="2026-11-02 04:00:02.170226525 +0000 UTC m=+4344.165927529" Nov 02 04:00:02.282526 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.282477 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" podStartSLOduration=2.282463762 podStartE2EDuration="2.282463762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:02.27904516 +0000 UTC m=+4344.274746104" watchObservedRunningTime="2026-11-02 04:00:02.282463762 +0000 UTC m=+4344.278164702" Nov 02 04:00:02.391593 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.391545 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" podStartSLOduration=2.39153011 podStartE2EDuration="2.39153011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:02.389764778 +0000 UTC m=+4344.385465721" watchObservedRunningTime="2026-11-02 04:00:02.39153011 +0000 UTC m=+4344.387231053" Nov 02 04:00:02.503882 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.503829 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" podStartSLOduration=2.5038133780000003 podStartE2EDuration="2.503813378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:02.501237946 +0000 UTC m=+4344.496938889" watchObservedRunningTime="2026-11-02 04:00:02.503813378 +0000 UTC m=+4344.499514322" Nov 02 04:00:02.739070 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.738995 4868 generic.go:356] "Generic (PLEG): container finished" podID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerID="498b86b40b379d80b8a2465c7eafd57fd41fe00ff2d81d975d3c31296c222282" exitCode=60 Nov 02 04:00:02.739225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.739069 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerDied","Data":"498b86b40b379d80b8a2465c7eafd57fd41fe00ff2d81d975d3c31296c222282"} Nov 02 04:00:02.739337 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.739316 4868 scope.go:122] "RemoveContainer" containerID="498b86b40b379d80b8a2465c7eafd57fd41fe00ff2d81d975d3c31296c222282" Nov 02 04:00:02.740747 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.740722 4868 generic.go:356] "Generic (PLEG): container finished" podID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerID="aa50c4a1136497977e76103800d63b53475a6ce9f17c44a757cd0f29b02839c3" exitCode=60 Nov 02 04:00:02.740849 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.740789 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerDied","Data":"aa50c4a1136497977e76103800d63b53475a6ce9f17c44a757cd0f29b02839c3"} Nov 02 04:00:02.741037 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.741016 4868 scope.go:122] "RemoveContainer" containerID="aa50c4a1136497977e76103800d63b53475a6ce9f17c44a757cd0f29b02839c3" Nov 02 04:00:02.742733 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.742639 4868 generic.go:356] "Generic (PLEG): container finished" podID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerID="06dc054da5fe4e688246d54909acfc8a37089a63c0b6477093976177ed6b4477" exitCode=60 Nov 02 04:00:02.742733 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.742705 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerDied","Data":"06dc054da5fe4e688246d54909acfc8a37089a63c0b6477093976177ed6b4477"} Nov 02 04:00:02.742975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.742958 4868 scope.go:122] "RemoveContainer" containerID="06dc054da5fe4e688246d54909acfc8a37089a63c0b6477093976177ed6b4477" Nov 02 04:00:02.744374 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.744331 4868 generic.go:356] "Generic (PLEG): container finished" podID="11967c4f-0f40-4152-8e76-2f5648550693" containerID="e018205305bcb561af1b393ae4f54e293a54b41a2311d98379a3d55eb634455d" exitCode=60 Nov 02 04:00:02.744461 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.744415 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerDied","Data":"e018205305bcb561af1b393ae4f54e293a54b41a2311d98379a3d55eb634455d"} Nov 02 04:00:02.744667 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.744645 4868 scope.go:122] "RemoveContainer" containerID="e018205305bcb561af1b393ae4f54e293a54b41a2311d98379a3d55eb634455d" Nov 02 04:00:02.746031 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.746010 4868 generic.go:356] "Generic (PLEG): container finished" podID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerID="0900f13caebdc67138a5a697c6af03f93138a7dae929ff7c34b9f7d5613d6987" exitCode=60 Nov 02 04:00:02.746119 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.746085 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerDied","Data":"0900f13caebdc67138a5a697c6af03f93138a7dae929ff7c34b9f7d5613d6987"} Nov 02 04:00:02.746272 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.746256 4868 scope.go:122] "RemoveContainer" containerID="0900f13caebdc67138a5a697c6af03f93138a7dae929ff7c34b9f7d5613d6987" Nov 02 04:00:02.747720 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.747700 4868 generic.go:356] "Generic (PLEG): container finished" podID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerID="88aaea75bb84c23b61094ce904bd286e885a4d5cb4b6fac43553dc489034ade8" exitCode=60 Nov 02 04:00:02.747807 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.747785 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerDied","Data":"88aaea75bb84c23b61094ce904bd286e885a4d5cb4b6fac43553dc489034ade8"} Nov 02 04:00:02.747951 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.747936 4868 scope.go:122] "RemoveContainer" containerID="88aaea75bb84c23b61094ce904bd286e885a4d5cb4b6fac43553dc489034ade8" Nov 02 04:00:02.749499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.749481 4868 generic.go:356] "Generic (PLEG): container finished" podID="09169936-e489-4c22-9d56-8478c2436f68" containerID="c12ef6cbdb7fb36eee82556152b3dea62ce7706604296deee5b54ab5196d1ae5" exitCode=60 Nov 02 04:00:02.749602 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.749550 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerDied","Data":"c12ef6cbdb7fb36eee82556152b3dea62ce7706604296deee5b54ab5196d1ae5"} Nov 02 04:00:02.749812 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.749792 4868 scope.go:122] "RemoveContainer" containerID="c12ef6cbdb7fb36eee82556152b3dea62ce7706604296deee5b54ab5196d1ae5" Nov 02 04:00:02.753875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.753856 4868 generic.go:356] "Generic (PLEG): container finished" podID="a92b6429-925b-420d-b916-fdf9f30e979c" containerID="b1835aa521fe2d085e6505de27bcaa011401b4a1c0c20bd71a24676609ff4f0d" exitCode=60 Nov 02 04:00:02.753959 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.753947 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerDied","Data":"b1835aa521fe2d085e6505de27bcaa011401b4a1c0c20bd71a24676609ff4f0d"} Nov 02 04:00:02.754172 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.754158 4868 scope.go:122] "RemoveContainer" containerID="b1835aa521fe2d085e6505de27bcaa011401b4a1c0c20bd71a24676609ff4f0d" Nov 02 04:00:02.755771 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.755754 4868 generic.go:356] "Generic (PLEG): container finished" podID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerID="f4bafb9baf43922f250d9fec03144293b8d84a927999d19ac81c9c5ecf482437" exitCode=60 Nov 02 04:00:02.755839 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.755805 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerDied","Data":"f4bafb9baf43922f250d9fec03144293b8d84a927999d19ac81c9c5ecf482437"} Nov 02 04:00:02.756078 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.756058 4868 scope.go:122] "RemoveContainer" containerID="f4bafb9baf43922f250d9fec03144293b8d84a927999d19ac81c9c5ecf482437" Nov 02 04:00:02.759853 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.759831 4868 generic.go:356] "Generic (PLEG): container finished" podID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerID="5e88d5675af913db530d942d8f08eee76107c0da9ba4ed14eb6b2415ac09d771" exitCode=60 Nov 02 04:00:02.759957 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.759930 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerDied","Data":"5e88d5675af913db530d942d8f08eee76107c0da9ba4ed14eb6b2415ac09d771"} Nov 02 04:00:02.760223 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.760195 4868 scope.go:122] "RemoveContainer" containerID="5e88d5675af913db530d942d8f08eee76107c0da9ba4ed14eb6b2415ac09d771" Nov 02 04:00:02.762820 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.762797 4868 generic.go:356] "Generic (PLEG): container finished" podID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerID="913af3404be10b51596f51933133b66bee3709f73452f94e187fcc02f99120d6" exitCode=60 Nov 02 04:00:02.762949 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.762878 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerDied","Data":"913af3404be10b51596f51933133b66bee3709f73452f94e187fcc02f99120d6"} Nov 02 04:00:02.764078 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:02.764053 4868 scope.go:122] "RemoveContainer" containerID="913af3404be10b51596f51933133b66bee3709f73452f94e187fcc02f99120d6" Nov 02 04:00:03.452448 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:03.452412 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:00:03.771489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.771458 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerStarted","Data":"352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a"} Nov 02 04:00:03.773406 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.773379 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerStarted","Data":"89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26"} Nov 02 04:00:03.775149 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.775123 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerStarted","Data":"1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91"} Nov 02 04:00:03.776951 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.776931 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerStarted","Data":"906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59"} Nov 02 04:00:03.778775 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.778754 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerStarted","Data":"a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa"} Nov 02 04:00:03.780467 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.780446 4868 generic.go:356] "Generic (PLEG): container finished" podID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerID="2a1895093b5a315bd4d3fbfa0443c4d77d85900eadfa3fb027f8f72246fd205c" exitCode=60 Nov 02 04:00:03.780584 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.780527 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerDied","Data":"2a1895093b5a315bd4d3fbfa0443c4d77d85900eadfa3fb027f8f72246fd205c"} Nov 02 04:00:03.780584 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.780577 4868 scope.go:122] "RemoveContainer" containerID="06dc054da5fe4e688246d54909acfc8a37089a63c0b6477093976177ed6b4477" Nov 02 04:00:03.780712 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.780694 4868 scope.go:122] "RemoveContainer" containerID="2a1895093b5a315bd4d3fbfa0443c4d77d85900eadfa3fb027f8f72246fd205c" Nov 02 04:00:03.780989 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:03.780964 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c_odh-ai-gateway-infra(f0e19ea6-be5d-459a-a313-cb6b8d925fd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" Nov 02 04:00:03.782346 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.782324 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerStarted","Data":"028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a"} Nov 02 04:00:03.784053 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.784028 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerStarted","Data":"8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c"} Nov 02 04:00:03.786040 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.786017 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerStarted","Data":"77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd"} Nov 02 04:00:03.788002 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.787977 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerStarted","Data":"b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed"} Nov 02 04:00:03.789738 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:03.789711 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerStarted","Data":"9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3"} Nov 02 04:00:04.082747 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.082701 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" podStartSLOduration=4.08268763 podStartE2EDuration="4.08268763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:04.079871204 +0000 UTC m=+4346.075572148" watchObservedRunningTime="2026-11-02 04:00:04.08268763 +0000 UTC m=+4346.078388573" Nov 02 04:00:04.415297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.415244 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" podStartSLOduration=4.415227546 podStartE2EDuration="4.415227546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:04.413053374 +0000 UTC m=+4346.408754317" watchObservedRunningTime="2026-11-02 04:00:04.415227546 +0000 UTC m=+4346.410928488" Nov 02 04:00:04.798592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.798554 4868 generic.go:356] "Generic (PLEG): container finished" podID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerID="352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a" exitCode=60 Nov 02 04:00:04.799059 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.798640 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerDied","Data":"352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a"} Nov 02 04:00:04.799059 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.798699 4868 scope.go:122] "RemoveContainer" containerID="f4bafb9baf43922f250d9fec03144293b8d84a927999d19ac81c9c5ecf482437" Nov 02 04:00:04.799059 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.798888 4868 scope.go:122] "RemoveContainer" containerID="352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a" Nov 02 04:00:04.799203 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.799186 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz_odh-ai-gateway-infra(09fe7ef7-fb83-47d1-89db-1b9d898bae39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" Nov 02 04:00:04.800794 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.800772 4868 generic.go:356] "Generic (PLEG): container finished" podID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerID="89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26" exitCode=60 Nov 02 04:00:04.800921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.800854 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerDied","Data":"89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26"} Nov 02 04:00:04.801067 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.801050 4868 scope.go:122] "RemoveContainer" containerID="89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26" Nov 02 04:00:04.801321 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.801298 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp_odh-ai-gateway-infra(d7258881-8b2d-4a65-8dc6-e59d2d13701c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" Nov 02 04:00:04.802830 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.802809 4868 generic.go:356] "Generic (PLEG): container finished" podID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerID="1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91" exitCode=60 Nov 02 04:00:04.802942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.802881 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerDied","Data":"1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91"} Nov 02 04:00:04.803067 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.803052 4868 scope.go:122] "RemoveContainer" containerID="1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91" Nov 02 04:00:04.803290 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.803267 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv_odh-ai-gateway-infra(3760203b-bcac-4f6a-a50b-9943eedb32aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" Nov 02 04:00:04.804759 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.804737 4868 generic.go:356] "Generic (PLEG): container finished" podID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerID="906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59" exitCode=60 Nov 02 04:00:04.804846 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.804802 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerDied","Data":"906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59"} Nov 02 04:00:04.804974 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.804956 4868 scope.go:122] "RemoveContainer" containerID="906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59" Nov 02 04:00:04.805176 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.805152 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n_odh-ai-gateway-infra(e85e0a2b-8ce6-4b15-bb89-d017abfb072d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" Nov 02 04:00:04.806794 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.806773 4868 generic.go:356] "Generic (PLEG): container finished" podID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerID="a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa" exitCode=60 Nov 02 04:00:04.806886 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.806800 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerDied","Data":"a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa"} Nov 02 04:00:04.807043 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.807016 4868 scope.go:122] "RemoveContainer" containerID="a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa" Nov 02 04:00:04.807214 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.807193 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4_odh-ai-gateway-infra(ade750b7-8be1-45da-aeb2-cfd2e6cef52b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" Nov 02 04:00:04.812944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.812923 4868 scope.go:122] "RemoveContainer" containerID="2a1895093b5a315bd4d3fbfa0443c4d77d85900eadfa3fb027f8f72246fd205c" Nov 02 04:00:04.813149 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.813128 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c_odh-ai-gateway-infra(f0e19ea6-be5d-459a-a313-cb6b8d925fd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" Nov 02 04:00:04.814918 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.814897 4868 generic.go:356] "Generic (PLEG): container finished" podID="11967c4f-0f40-4152-8e76-2f5648550693" containerID="028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a" exitCode=60 Nov 02 04:00:04.815034 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.814961 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerDied","Data":"028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a"} Nov 02 04:00:04.815207 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.815132 4868 scope.go:122] "RemoveContainer" containerID="028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a" Nov 02 04:00:04.815309 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.815284 4868 scope.go:122] "RemoveContainer" containerID="5e88d5675af913db530d942d8f08eee76107c0da9ba4ed14eb6b2415ac09d771" Nov 02 04:00:04.815405 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.815313 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds_odh-ai-gateway-infra(11967c4f-0f40-4152-8e76-2f5648550693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" podUID="11967c4f-0f40-4152-8e76-2f5648550693" Nov 02 04:00:04.817039 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.817019 4868 generic.go:356] "Generic (PLEG): container finished" podID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerID="8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c" exitCode=60 Nov 02 04:00:04.817122 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.817089 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerDied","Data":"8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c"} Nov 02 04:00:04.817243 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.817229 4868 scope.go:122] "RemoveContainer" containerID="8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c" Nov 02 04:00:04.817425 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.817406 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6_odh-ai-gateway-infra(294241de-b2e0-4c92-af80-dfdbaa6060c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" Nov 02 04:00:04.818845 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.818828 4868 generic.go:356] "Generic (PLEG): container finished" podID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerID="77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd" exitCode=60 Nov 02 04:00:04.818940 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.818908 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerDied","Data":"77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd"} Nov 02 04:00:04.819052 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.819039 4868 scope.go:122] "RemoveContainer" containerID="77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd" Nov 02 04:00:04.819224 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.819208 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893200-4lxjf_odh-ai-gateway-infra(4377c8b4-e5d5-4192-a130-0b7385acd12e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" Nov 02 04:00:04.820601 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.820581 4868 generic.go:356] "Generic (PLEG): container finished" podID="09169936-e489-4c22-9d56-8478c2436f68" containerID="b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed" exitCode=60 Nov 02 04:00:04.820691 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.820660 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerDied","Data":"b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed"} Nov 02 04:00:04.820832 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.820816 4868 scope.go:122] "RemoveContainer" containerID="b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed" Nov 02 04:00:04.821018 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.820998 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp_odh-ai-gateway-infra(09169936-e489-4c22-9d56-8478c2436f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" podUID="09169936-e489-4c22-9d56-8478c2436f68" Nov 02 04:00:04.822384 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.822362 4868 generic.go:356] "Generic (PLEG): container finished" podID="a92b6429-925b-420d-b916-fdf9f30e979c" containerID="9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3" exitCode=60 Nov 02 04:00:04.822490 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.822410 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerDied","Data":"9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3"} Nov 02 04:00:04.822638 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.822620 4868 scope.go:122] "RemoveContainer" containerID="9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3" Nov 02 04:00:04.822820 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:04.822801 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q_odh-ai-gateway-infra(a92b6429-925b-420d-b916-fdf9f30e979c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" Nov 02 04:00:04.828875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.828822 4868 scope.go:122] "RemoveContainer" containerID="913af3404be10b51596f51933133b66bee3709f73452f94e187fcc02f99120d6" Nov 02 04:00:04.841712 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.841697 4868 scope.go:122] "RemoveContainer" containerID="498b86b40b379d80b8a2465c7eafd57fd41fe00ff2d81d975d3c31296c222282" Nov 02 04:00:04.855009 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.854987 4868 scope.go:122] "RemoveContainer" containerID="aa50c4a1136497977e76103800d63b53475a6ce9f17c44a757cd0f29b02839c3" Nov 02 04:00:04.860542 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.860504 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" podStartSLOduration=4.860492512 podStartE2EDuration="4.860492512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:04.858665784 +0000 UTC m=+4346.854366724" watchObservedRunningTime="2026-11-02 04:00:04.860492512 +0000 UTC m=+4346.856193454" Nov 02 04:00:04.868849 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.868830 4868 scope.go:122] "RemoveContainer" containerID="e018205305bcb561af1b393ae4f54e293a54b41a2311d98379a3d55eb634455d" Nov 02 04:00:04.882566 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.882548 4868 scope.go:122] "RemoveContainer" containerID="0900f13caebdc67138a5a697c6af03f93138a7dae929ff7c34b9f7d5613d6987" Nov 02 04:00:04.897232 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.897212 4868 scope.go:122] "RemoveContainer" containerID="88aaea75bb84c23b61094ce904bd286e885a4d5cb4b6fac43553dc489034ade8" Nov 02 04:00:04.957661 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.936281 4868 scope.go:122] "RemoveContainer" containerID="c12ef6cbdb7fb36eee82556152b3dea62ce7706604296deee5b54ab5196d1ae5" Nov 02 04:00:04.986088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:04.985732 4868 scope.go:122] "RemoveContainer" containerID="b1835aa521fe2d085e6505de27bcaa011401b4a1c0c20bd71a24676609ff4f0d" Nov 02 04:00:05.101561 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.101506 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" podStartSLOduration=5.101487436 podStartE2EDuration="5.101487436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:05.096853227 +0000 UTC m=+4347.092554169" watchObservedRunningTime="2026-11-02 04:00:05.101487436 +0000 UTC m=+4347.097188380" Nov 02 04:00:05.836293 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.836261 4868 scope.go:122] "RemoveContainer" containerID="028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a" Nov 02 04:00:05.836831 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.836619 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds_odh-ai-gateway-infra(11967c4f-0f40-4152-8e76-2f5648550693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" podUID="11967c4f-0f40-4152-8e76-2f5648550693" Nov 02 04:00:05.839015 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.838988 4868 scope.go:122] "RemoveContainer" containerID="8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c" Nov 02 04:00:05.839334 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.839307 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6_odh-ai-gateway-infra(294241de-b2e0-4c92-af80-dfdbaa6060c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" Nov 02 04:00:05.847949 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.847913 4868 scope.go:122] "RemoveContainer" containerID="77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd" Nov 02 04:00:05.848209 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.848188 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893200-4lxjf_odh-ai-gateway-infra(4377c8b4-e5d5-4192-a130-0b7385acd12e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" Nov 02 04:00:05.850541 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.850517 4868 scope.go:122] "RemoveContainer" containerID="b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed" Nov 02 04:00:05.850853 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.850826 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp_odh-ai-gateway-infra(09169936-e489-4c22-9d56-8478c2436f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" podUID="09169936-e489-4c22-9d56-8478c2436f68" Nov 02 04:00:05.852232 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.852209 4868 scope.go:122] "RemoveContainer" containerID="9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3" Nov 02 04:00:05.852562 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.852535 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q_odh-ai-gateway-infra(a92b6429-925b-420d-b916-fdf9f30e979c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" Nov 02 04:00:05.853812 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.853795 4868 scope.go:122] "RemoveContainer" containerID="352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a" Nov 02 04:00:05.854062 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.854041 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz_odh-ai-gateway-infra(09fe7ef7-fb83-47d1-89db-1b9d898bae39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" Nov 02 04:00:05.855261 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.855244 4868 scope.go:122] "RemoveContainer" containerID="89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26" Nov 02 04:00:05.855499 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.855481 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp_odh-ai-gateway-infra(d7258881-8b2d-4a65-8dc6-e59d2d13701c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" Nov 02 04:00:05.856736 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.856719 4868 scope.go:122] "RemoveContainer" containerID="1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91" Nov 02 04:00:05.856949 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.856927 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv_odh-ai-gateway-infra(3760203b-bcac-4f6a-a50b-9943eedb32aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" Nov 02 04:00:05.858217 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.858202 4868 scope.go:122] "RemoveContainer" containerID="906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59" Nov 02 04:00:05.858447 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.858421 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n_odh-ai-gateway-infra(e85e0a2b-8ce6-4b15-bb89-d017abfb072d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" Nov 02 04:00:05.859659 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:05.859644 4868 scope.go:122] "RemoveContainer" containerID="a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa" Nov 02 04:00:05.859818 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:05.859803 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4_odh-ai-gateway-infra(ade750b7-8be1-45da-aeb2-cfd2e6cef52b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" Nov 02 04:00:15.108660 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.108627 4868 scope.go:122] "RemoveContainer" containerID="77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd" Nov 02 04:00:15.109052 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.108678 4868 scope.go:122] "RemoveContainer" containerID="2a1895093b5a315bd4d3fbfa0443c4d77d85900eadfa3fb027f8f72246fd205c" Nov 02 04:00:15.935228 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.935195 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerStarted","Data":"7ddb93f132ec74a6d057d56c520d9e9fbb07a1be9a5f8a902cb0f57973091ac1"} Nov 02 04:00:15.937027 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.937003 4868 generic.go:356] "Generic (PLEG): container finished" podID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerID="3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068" exitCode=0 Nov 02 04:00:15.937154 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.937075 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" event={"ID":"24607332-f862-4be8-9712-8e1e7f6c1b68","Type":"ContainerDied","Data":"3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068"} Nov 02 04:00:15.937436 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.937417 4868 scope.go:122] "RemoveContainer" containerID="3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068" Nov 02 04:00:15.939213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.939193 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerStarted","Data":"51696256e345625627ec4315d01fb3e0da15fdd8dc36eb44bcd5177e2c1da98d"} Nov 02 04:00:15.939304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.939226 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:00:15.939304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.939289 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:00:15.939414 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:15.939322 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:00:16.108279 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.108250 4868 scope.go:122] "RemoveContainer" containerID="906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59" Nov 02 04:00:16.108817 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.108524 4868 scope.go:122] "RemoveContainer" containerID="352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a" Nov 02 04:00:16.108817 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.108666 4868 scope.go:122] "RemoveContainer" containerID="9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3" Nov 02 04:00:16.108817 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.108766 4868 scope.go:122] "RemoveContainer" containerID="89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26" Nov 02 04:00:16.963640 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.963607 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerStarted","Data":"7e1b86f3d76183bbda1222590fb128d5bc5fdf0fd7d1e031d485e83c539924bf"} Nov 02 04:00:16.966009 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.965989 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerStarted","Data":"c7678515c0bfa6b75918b922d6096f93399b175c564455aa7a5d2e646ea2d56b"} Nov 02 04:00:16.967604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.967582 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerStarted","Data":"a1c3edf27ace83d23a5643f798184c3344ab1a004c5a56a290e07eaca1758462"} Nov 02 04:00:16.969287 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.969263 4868 generic.go:356] "Generic (PLEG): container finished" podID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerID="51696256e345625627ec4315d01fb3e0da15fdd8dc36eb44bcd5177e2c1da98d" exitCode=60 Nov 02 04:00:16.969339 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.969320 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerDied","Data":"51696256e345625627ec4315d01fb3e0da15fdd8dc36eb44bcd5177e2c1da98d"} Nov 02 04:00:16.969404 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.969376 4868 scope.go:122] "RemoveContainer" containerID="2a1895093b5a315bd4d3fbfa0443c4d77d85900eadfa3fb027f8f72246fd205c" Nov 02 04:00:16.969631 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.969618 4868 scope.go:122] "RemoveContainer" containerID="51696256e345625627ec4315d01fb3e0da15fdd8dc36eb44bcd5177e2c1da98d" Nov 02 04:00:16.969830 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:16.969813 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c_odh-ai-gateway-infra(f0e19ea6-be5d-459a-a313-cb6b8d925fd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" Nov 02 04:00:16.971542 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.971522 4868 generic.go:356] "Generic (PLEG): container finished" podID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerID="7ddb93f132ec74a6d057d56c520d9e9fbb07a1be9a5f8a902cb0f57973091ac1" exitCode=60 Nov 02 04:00:16.971600 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.971584 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerDied","Data":"7ddb93f132ec74a6d057d56c520d9e9fbb07a1be9a5f8a902cb0f57973091ac1"} Nov 02 04:00:16.971851 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.971831 4868 scope.go:122] "RemoveContainer" containerID="7ddb93f132ec74a6d057d56c520d9e9fbb07a1be9a5f8a902cb0f57973091ac1" Nov 02 04:00:16.972051 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:16.972034 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893200-4lxjf_odh-ai-gateway-infra(4377c8b4-e5d5-4192-a130-0b7385acd12e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" Nov 02 04:00:16.973827 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.973805 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" event={"ID":"24607332-f862-4be8-9712-8e1e7f6c1b68","Type":"ContainerStarted","Data":"f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753"} Nov 02 04:00:16.975860 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.975841 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerStarted","Data":"025ddf659279ca1a24fb26442c267f3c2f1d82f1ec0ba229a0869411a85cf242"} Nov 02 04:00:16.987567 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:16.984560 4868 scope.go:122] "RemoveContainer" containerID="77d5d3fd1dbc82f18819745333b299f1939451956d35c387ca408e8420918acd" Nov 02 04:00:17.064267 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.064221 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf"] Nov 02 04:00:17.107923 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.107890 4868 scope.go:122] "RemoveContainer" containerID="1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91" Nov 02 04:00:17.177604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.177575 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c"] Nov 02 04:00:17.752287 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.749195 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 04:00:17.752504 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.752327 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 04:00:17.756673 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.756442 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 04:00:17.986396 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.986347 4868 generic.go:356] "Generic (PLEG): container finished" podID="a92b6429-925b-420d-b916-fdf9f30e979c" containerID="025ddf659279ca1a24fb26442c267f3c2f1d82f1ec0ba229a0869411a85cf242" exitCode=60 Nov 02 04:00:17.986396 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.986384 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerDied","Data":"025ddf659279ca1a24fb26442c267f3c2f1d82f1ec0ba229a0869411a85cf242"} Nov 02 04:00:17.986634 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.986442 4868 scope.go:122] "RemoveContainer" containerID="9c3dbd36ee3cb15016d8cc2a5d17be605333ba1192d6903378da80590b9750a3" Nov 02 04:00:17.986836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.986808 4868 scope.go:122] "RemoveContainer" containerID="025ddf659279ca1a24fb26442c267f3c2f1d82f1ec0ba229a0869411a85cf242" Nov 02 04:00:17.987180 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:17.987152 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q_odh-ai-gateway-infra(a92b6429-925b-420d-b916-fdf9f30e979c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" Nov 02 04:00:17.988672 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.988652 4868 generic.go:356] "Generic (PLEG): container finished" podID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerID="7e1b86f3d76183bbda1222590fb128d5bc5fdf0fd7d1e031d485e83c539924bf" exitCode=60 Nov 02 04:00:17.988791 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.988720 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerDied","Data":"7e1b86f3d76183bbda1222590fb128d5bc5fdf0fd7d1e031d485e83c539924bf"} Nov 02 04:00:17.989125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.989106 4868 scope.go:122] "RemoveContainer" containerID="7e1b86f3d76183bbda1222590fb128d5bc5fdf0fd7d1e031d485e83c539924bf" Nov 02 04:00:17.989421 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:17.989399 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz_odh-ai-gateway-infra(09fe7ef7-fb83-47d1-89db-1b9d898bae39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" Nov 02 04:00:17.990800 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.990770 4868 generic.go:356] "Generic (PLEG): container finished" podID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerID="c7678515c0bfa6b75918b922d6096f93399b175c564455aa7a5d2e646ea2d56b" exitCode=60 Nov 02 04:00:17.990919 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.990840 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerDied","Data":"c7678515c0bfa6b75918b922d6096f93399b175c564455aa7a5d2e646ea2d56b"} Nov 02 04:00:17.991191 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.991176 4868 scope.go:122] "RemoveContainer" containerID="c7678515c0bfa6b75918b922d6096f93399b175c564455aa7a5d2e646ea2d56b" Nov 02 04:00:17.991493 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:17.991471 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp_odh-ai-gateway-infra(d7258881-8b2d-4a65-8dc6-e59d2d13701c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" Nov 02 04:00:17.993077 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.993054 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerStarted","Data":"e11ab8ac7b65d4701c76c4c2ceebb5f349a6c818908de27cb2f9eef1224bc41a"} Nov 02 04:00:17.995175 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.995156 4868 generic.go:356] "Generic (PLEG): container finished" podID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerID="a1c3edf27ace83d23a5643f798184c3344ab1a004c5a56a290e07eaca1758462" exitCode=60 Nov 02 04:00:17.995252 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.995232 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerDied","Data":"a1c3edf27ace83d23a5643f798184c3344ab1a004c5a56a290e07eaca1758462"} Nov 02 04:00:17.995549 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:17.995531 4868 scope.go:122] "RemoveContainer" containerID="a1c3edf27ace83d23a5643f798184c3344ab1a004c5a56a290e07eaca1758462" Nov 02 04:00:17.995806 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:17.995786 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n_odh-ai-gateway-infra(e85e0a2b-8ce6-4b15-bb89-d017abfb072d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" Nov 02 04:00:18.001098 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.001078 4868 scope.go:122] "RemoveContainer" containerID="352c09a9da85b74136cb8afe13ef900cbff4f4bdf6e524620e59bef9fb257b9a" Nov 02 04:00:18.017909 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.017888 4868 scope.go:122] "RemoveContainer" containerID="89edf3f623790320f364d83b5ee9f48bf4d32a0bc2349c6dd3a47b69f06cce26" Nov 02 04:00:18.048601 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.039624 4868 scope.go:122] "RemoveContainer" containerID="906e4fdc36d37440bb30bd1b07e98dc0164b5a1f8a672985340e8f0e0c91eb59" Nov 02 04:00:18.098935 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.098138 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz"] Nov 02 04:00:18.123072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.119288 4868 scope.go:122] "RemoveContainer" containerID="028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a" Nov 02 04:00:18.123072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.120109 4868 scope.go:122] "RemoveContainer" containerID="8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c" Nov 02 04:00:18.123072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.121002 4868 scope.go:122] "RemoveContainer" containerID="a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa" Nov 02 04:00:18.128643 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.125999 4868 scope.go:122] "RemoveContainer" containerID="b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed" Nov 02 04:00:18.400268 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.399125 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n"] Nov 02 04:00:18.475697 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.475617 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:18.481625 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.481603 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:18.576852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.576768 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc\" (UniqueName: \"kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc\") pod \"4377c8b4-e5d5-4192-a130-0b7385acd12e\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " Nov 02 04:00:18.577178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.576854 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca\") pod \"4377c8b4-e5d5-4192-a130-0b7385acd12e\" (UID: \"4377c8b4-e5d5-4192-a130-0b7385acd12e\") " Nov 02 04:00:18.577178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.576912 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77\" (UniqueName: \"kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77\") pod \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " Nov 02 04:00:18.577178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.577011 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca\") pod \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\" (UID: \"f0e19ea6-be5d-459a-a313-cb6b8d925fd3\") " Nov 02 04:00:18.577178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.577157 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca" pod "4377c8b4-e5d5-4192-a130-0b7385acd12e" (UID: "4377c8b4-e5d5-4192-a130-0b7385acd12e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:18.577542 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.577524 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c8b4-e5d5-4192-a130-0b7385acd12e-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:18.577607 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.577577 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca" pod "f0e19ea6-be5d-459a-a313-cb6b8d925fd3" (UID: "f0e19ea6-be5d-459a-a313-cb6b8d925fd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:18.659372 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.658544 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77" pod "f0e19ea6-be5d-459a-a313-cb6b8d925fd3" (UID: "f0e19ea6-be5d-459a-a313-cb6b8d925fd3"). InnerVolumeSpecName "kube-api-access-ppd77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:18.660591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.660563 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc" pod "4377c8b4-e5d5-4192-a130-0b7385acd12e" (UID: "4377c8b4-e5d5-4192-a130-0b7385acd12e"). InnerVolumeSpecName "kube-api-access-cpmlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:18.662174 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.662147 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp"] Nov 02 04:00:18.678763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.678740 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:18.678763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.678764 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmlc\" (UniqueName: \"kubernetes.io/projected/4377c8b4-e5d5-4192-a130-0b7385acd12e-kube-api-access-cpmlc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:18.678888 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.678774 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppd77\" (UniqueName: \"kubernetes.io/projected/f0e19ea6-be5d-459a-a313-cb6b8d925fd3-kube-api-access-ppd77\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:18.751794 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.751757 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 04:00:18.752862 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.752830 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 04:00:18.764679 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.764655 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q"] Nov 02 04:00:18.862936 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:18.862906 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 04:00:19.014651 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.014617 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerStarted","Data":"28ab160f99d597ccb9033e6dd403fdba496a53b638a83fe61bcbb3c2ff467c83"} Nov 02 04:00:19.016469 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.016442 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" event={"ID":"f0e19ea6-be5d-459a-a313-cb6b8d925fd3","Type":"ContainerDied","Data":"a2953f9d00da532211740b15285dd58c9c065d6e1741a5cea5ef14341fa6db92"} Nov 02 04:00:19.016627 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.016505 4868 scope.go:122] "RemoveContainer" containerID="51696256e345625627ec4315d01fb3e0da15fdd8dc36eb44bcd5177e2c1da98d" Nov 02 04:00:19.016627 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.016452 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c" Nov 02 04:00:19.028853 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.028830 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerStarted","Data":"0b87ea7f04377d72f6ab5e5356da900adaa7ca1e0aaae06d4b566f2b74246750"} Nov 02 04:00:19.063926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.063154 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerStarted","Data":"bb1fff225302174d395ced7b654815ef2658e60016772802c446d803aef10df4"} Nov 02 04:00:19.066268 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.066240 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" event={"ID":"4377c8b4-e5d5-4192-a130-0b7385acd12e","Type":"ContainerDied","Data":"264e8a3f350c7c9d54fa69cc6ce68e901fdbc609b65bc86bd3a65fdeaa7a71b0"} Nov 02 04:00:19.066493 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.066476 4868 scope.go:122] "RemoveContainer" containerID="7ddb93f132ec74a6d057d56c520d9e9fbb07a1be9a5f8a902cb0f57973091ac1" Nov 02 04:00:19.066717 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.066702 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf" Nov 02 04:00:19.075688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.074937 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerStarted","Data":"15191fcf8ad71a8cbe346fa48190d92896c835a3ff4f91a815a1425210c9d303"} Nov 02 04:00:19.090757 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.090733 4868 generic.go:356] "Generic (PLEG): container finished" podID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerID="e11ab8ac7b65d4701c76c4c2ceebb5f349a6c818908de27cb2f9eef1224bc41a" exitCode=60 Nov 02 04:00:19.091330 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.091298 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerDied","Data":"e11ab8ac7b65d4701c76c4c2ceebb5f349a6c818908de27cb2f9eef1224bc41a"} Nov 02 04:00:19.091455 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.091368 4868 scope.go:122] "RemoveContainer" containerID="1c9c9d5dc4a19af6fc148b70d1e1db3fd94039f15758e48567ef8a8b218d6d91" Nov 02 04:00:19.154014 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.110479 4868 scope.go:122] "RemoveContainer" containerID="e11ab8ac7b65d4701c76c4c2ceebb5f349a6c818908de27cb2f9eef1224bc41a" Nov 02 04:00:19.154014 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:19.110920 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv_odh-ai-gateway-infra(3760203b-bcac-4f6a-a50b-9943eedb32aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" Nov 02 04:00:19.276178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.275684 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv"] Nov 02 04:00:19.313454 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.313422 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:19.397685 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.397657 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2\" (UniqueName: \"kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2\") pod \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " Nov 02 04:00:19.397900 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.397878 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca\") pod \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\" (UID: \"e85e0a2b-8ce6-4b15-bb89-d017abfb072d\") " Nov 02 04:00:19.398224 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.398200 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca" pod "e85e0a2b-8ce6-4b15-bb89-d017abfb072d" (UID: "e85e0a2b-8ce6-4b15-bb89-d017abfb072d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:19.398435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.398418 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:19.400765 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.400746 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2" pod "e85e0a2b-8ce6-4b15-bb89-d017abfb072d" (UID: "e85e0a2b-8ce6-4b15-bb89-d017abfb072d"). InnerVolumeSpecName "kube-api-access-vsld2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:19.477492 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.477440 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c"] Nov 02 04:00:19.481259 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.481231 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee0fed-29893200-qkh5c"] Nov 02 04:00:19.499980 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.499954 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsld2\" (UniqueName: \"kubernetes.io/projected/e85e0a2b-8ce6-4b15-bb89-d017abfb072d-kube-api-access-vsld2\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:19.756500 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.756455 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 04:00:19.756641 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.756509 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 04:00:19.794333 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.794309 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:19.798154 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.798135 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:19.801735 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.801710 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:19.904167 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904135 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca\") pod \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " Nov 02 04:00:19.904338 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904221 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk\" (UniqueName: \"kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk\") pod \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " Nov 02 04:00:19.904338 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904301 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca\") pod \"a92b6429-925b-420d-b916-fdf9f30e979c\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " Nov 02 04:00:19.904472 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904345 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c\" (UniqueName: \"kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c\") pod \"a92b6429-925b-420d-b916-fdf9f30e979c\" (UID: \"a92b6429-925b-420d-b916-fdf9f30e979c\") " Nov 02 04:00:19.904472 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904434 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm\" (UniqueName: \"kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm\") pod \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\" (UID: \"d7258881-8b2d-4a65-8dc6-e59d2d13701c\") " Nov 02 04:00:19.904472 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904468 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca\") pod \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\" (UID: \"09fe7ef7-fb83-47d1-89db-1b9d898bae39\") " Nov 02 04:00:19.904648 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904528 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca" pod "d7258881-8b2d-4a65-8dc6-e59d2d13701c" (UID: "d7258881-8b2d-4a65-8dc6-e59d2d13701c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:19.904713 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904663 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca" pod "a92b6429-925b-420d-b916-fdf9f30e979c" (UID: "a92b6429-925b-420d-b916-fdf9f30e979c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:19.904940 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904911 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca" pod "09fe7ef7-fb83-47d1-89db-1b9d898bae39" (UID: "09fe7ef7-fb83-47d1-89db-1b9d898bae39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:19.905070 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.904981 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7258881-8b2d-4a65-8dc6-e59d2d13701c-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:19.905070 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.905001 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92b6429-925b-420d-b916-fdf9f30e979c-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:19.906465 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.906443 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk" pod "09fe7ef7-fb83-47d1-89db-1b9d898bae39" (UID: "09fe7ef7-fb83-47d1-89db-1b9d898bae39"). InnerVolumeSpecName "kube-api-access-6ktqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:19.907028 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.907004 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c" pod "a92b6429-925b-420d-b916-fdf9f30e979c" (UID: "a92b6429-925b-420d-b916-fdf9f30e979c"). InnerVolumeSpecName "kube-api-access-6t56c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:19.907028 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:19.907014 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm" pod "d7258881-8b2d-4a65-8dc6-e59d2d13701c" (UID: "d7258881-8b2d-4a65-8dc6-e59d2d13701c"). InnerVolumeSpecName "kube-api-access-hnnqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:20.005963 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.005871 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t56c\" (UniqueName: \"kubernetes.io/projected/a92b6429-925b-420d-b916-fdf9f30e979c-kube-api-access-6t56c\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:20.005963 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.005895 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnnqm\" (UniqueName: \"kubernetes.io/projected/d7258881-8b2d-4a65-8dc6-e59d2d13701c-kube-api-access-hnnqm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:20.005963 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.005905 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09fe7ef7-fb83-47d1-89db-1b9d898bae39-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:20.005963 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.005916 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ktqk\" (UniqueName: \"kubernetes.io/projected/09fe7ef7-fb83-47d1-89db-1b9d898bae39-kube-api-access-6ktqk\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:20.099282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.099241 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" event={"ID":"09fe7ef7-fb83-47d1-89db-1b9d898bae39","Type":"ContainerDied","Data":"17fc43493c71613c07305943758bd9b6855f69b2e28cbcc069d48b84696c1530"} Nov 02 04:00:20.099501 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.099250 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz" Nov 02 04:00:20.099501 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.099301 4868 scope.go:122] "RemoveContainer" containerID="7e1b86f3d76183bbda1222590fb128d5bc5fdf0fd7d1e031d485e83c539924bf" Nov 02 04:00:20.100693 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.100671 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" Nov 02 04:00:20.100796 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.100690 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp" event={"ID":"d7258881-8b2d-4a65-8dc6-e59d2d13701c","Type":"ContainerDied","Data":"4690786171d640f65cb6677960dfe1fa4cd70babd37adec21cdb536d098c9fea"} Nov 02 04:00:20.104270 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.104245 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" event={"ID":"e85e0a2b-8ce6-4b15-bb89-d017abfb072d","Type":"ContainerDied","Data":"2367090f809033cb682d6272e3fd54ed3ce173e64aac5635c20d18576c0934b7"} Nov 02 04:00:20.104397 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.104331 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n" Nov 02 04:00:20.109681 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.109568 4868 generic.go:356] "Generic (PLEG): container finished" podID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerID="28ab160f99d597ccb9033e6dd403fdba496a53b638a83fe61bcbb3c2ff467c83" exitCode=60 Nov 02 04:00:20.123449 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.123364 4868 scope.go:122] "RemoveContainer" containerID="c7678515c0bfa6b75918b922d6096f93399b175c564455aa7a5d2e646ea2d56b" Nov 02 04:00:20.126632 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.126602 4868 generic.go:356] "Generic (PLEG): container finished" podID="11967c4f-0f40-4152-8e76-2f5648550693" containerID="0b87ea7f04377d72f6ab5e5356da900adaa7ca1e0aaae06d4b566f2b74246750" exitCode=60 Nov 02 04:00:20.129109 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.129085 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" path="/var/lib/kubelet/pods/f0e19ea6-be5d-459a-a313-cb6b8d925fd3/volumes" Nov 02 04:00:20.129936 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.129910 4868 generic.go:356] "Generic (PLEG): container finished" podID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerID="bb1fff225302174d395ced7b654815ef2658e60016772802c446d803aef10df4" exitCode=60 Nov 02 04:00:20.130224 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.130202 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerDied","Data":"28ab160f99d597ccb9033e6dd403fdba496a53b638a83fe61bcbb3c2ff467c83"} Nov 02 04:00:20.130425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.130408 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerDied","Data":"0b87ea7f04377d72f6ab5e5356da900adaa7ca1e0aaae06d4b566f2b74246750"} Nov 02 04:00:20.130559 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.130547 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerDied","Data":"bb1fff225302174d395ced7b654815ef2658e60016772802c446d803aef10df4"} Nov 02 04:00:20.130975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.130957 4868 scope.go:122] "RemoveContainer" containerID="28ab160f99d597ccb9033e6dd403fdba496a53b638a83fe61bcbb3c2ff467c83" Nov 02 04:00:20.131265 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.131249 4868 scope.go:122] "RemoveContainer" containerID="bb1fff225302174d395ced7b654815ef2658e60016772802c446d803aef10df4" Nov 02 04:00:20.156766 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:20.156736 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6_odh-ai-gateway-infra(294241de-b2e0-4c92-af80-dfdbaa6060c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" Nov 02 04:00:20.156959 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:20.139473 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4_odh-ai-gateway-infra(ade750b7-8be1-45da-aeb2-cfd2e6cef52b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" Nov 02 04:00:20.157067 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.157054 4868 scope.go:122] "RemoveContainer" containerID="0b87ea7f04377d72f6ab5e5356da900adaa7ca1e0aaae06d4b566f2b74246750" Nov 02 04:00:20.161983 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:20.157410 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds_odh-ai-gateway-infra(11967c4f-0f40-4152-8e76-2f5648550693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" podUID="11967c4f-0f40-4152-8e76-2f5648550693" Nov 02 04:00:20.165285 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.165264 4868 scope.go:122] "RemoveContainer" containerID="a1c3edf27ace83d23a5643f798184c3344ab1a004c5a56a290e07eaca1758462" Nov 02 04:00:20.166951 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.166926 4868 generic.go:356] "Generic (PLEG): container finished" podID="09169936-e489-4c22-9d56-8478c2436f68" containerID="15191fcf8ad71a8cbe346fa48190d92896c835a3ff4f91a815a1425210c9d303" exitCode=60 Nov 02 04:00:20.167062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.167019 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerDied","Data":"15191fcf8ad71a8cbe346fa48190d92896c835a3ff4f91a815a1425210c9d303"} Nov 02 04:00:20.167513 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.167427 4868 scope.go:122] "RemoveContainer" containerID="15191fcf8ad71a8cbe346fa48190d92896c835a3ff4f91a815a1425210c9d303" Nov 02 04:00:20.167808 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:20.167778 4868 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp_odh-ai-gateway-infra(09169936-e489-4c22-9d56-8478c2436f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" podUID="09169936-e489-4c22-9d56-8478c2436f68" Nov 02 04:00:20.172055 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.172029 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" event={"ID":"a92b6429-925b-420d-b916-fdf9f30e979c","Type":"ContainerDied","Data":"bb0fa68e02a3dc973f2608abd9eff39ec9b56ace603f6ac7394fe773a1805780"} Nov 02 04:00:20.172148 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.172137 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q" Nov 02 04:00:20.181207 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.181185 4868 scope.go:122] "RemoveContainer" containerID="a20266f86973e6c580b4e908c47cf88bfa25dab565915dd3e9254bd49ebfeafa" Nov 02 04:00:20.197678 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.197487 4868 scope.go:122] "RemoveContainer" containerID="028d6ce857a1f88791aff5597bf254d8fbdf21eb3682d0767d606834b038e84a" Nov 02 04:00:20.217467 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.217204 4868 scope.go:122] "RemoveContainer" containerID="8c969e29e63fefa13b03160588726d8e7856579a6b74ccfecfadd405df5a188c" Nov 02 04:00:20.268640 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.268627 4868 scope.go:122] "RemoveContainer" containerID="b0b3b3f7b849ef8961d5c922696d7144b39a528b15c8c1685fbc7555c770c0ed" Nov 02 04:00:20.289408 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.289367 4868 scope.go:122] "RemoveContainer" containerID="025ddf659279ca1a24fb26442c267f3c2f1d82f1ec0ba229a0869411a85cf242" Nov 02 04:00:20.290928 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.290908 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:20.362938 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.362902 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6"] Nov 02 04:00:20.411804 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.411775 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca\") pod \"3760203b-bcac-4f6a-a50b-9943eedb32aa\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " Nov 02 04:00:20.412297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.411813 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4\" (UniqueName: \"kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4\") pod \"3760203b-bcac-4f6a-a50b-9943eedb32aa\" (UID: \"3760203b-bcac-4f6a-a50b-9943eedb32aa\") " Nov 02 04:00:20.412297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.412195 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca" pod "3760203b-bcac-4f6a-a50b-9943eedb32aa" (UID: "3760203b-bcac-4f6a-a50b-9943eedb32aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:20.414050 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.414023 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4" pod "3760203b-bcac-4f6a-a50b-9943eedb32aa" (UID: "3760203b-bcac-4f6a-a50b-9943eedb32aa"). InnerVolumeSpecName "kube-api-access-bt2l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:20.513490 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.513452 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3760203b-bcac-4f6a-a50b-9943eedb32aa-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:20.513490 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.513487 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2l4\" (UniqueName: \"kubernetes.io/projected/3760203b-bcac-4f6a-a50b-9943eedb32aa-kube-api-access-bt2l4\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:20.581401 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.581306 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf"] Nov 02 04:00:20.586826 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.586799 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893200-4lxjf"] Nov 02 04:00:20.760378 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.759850 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 04:00:20.760378 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.759909 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 04:00:20.879343 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.879298 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n"] Nov 02 04:00:20.884336 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.884308 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d94cc3-29893200-wc42n"] Nov 02 04:00:20.959066 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:20.959034 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds"] Nov 02 04:00:21.059698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.059657 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz"] Nov 02 04:00:21.062986 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.062957 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ae1315-29893200-654vz"] Nov 02 04:00:21.081242 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.081216 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp"] Nov 02 04:00:21.197731 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.192890 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" event={"ID":"3760203b-bcac-4f6a-a50b-9943eedb32aa","Type":"ContainerDied","Data":"8d888ed83435fdf59703dca6ff1287e0a3073e5d1d673f7536a59ff9d7ae120c"} Nov 02 04:00:21.197731 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.192956 4868 scope.go:122] "RemoveContainer" containerID="e11ab8ac7b65d4701c76c4c2ceebb5f349a6c818908de27cb2f9eef1224bc41a" Nov 02 04:00:21.197731 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.193079 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv" Nov 02 04:00:21.271405 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.270841 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp"] Nov 02 04:00:21.280799 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.278288 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3b27a5-29893200-mtgrp"] Nov 02 04:00:21.365621 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.365582 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4"] Nov 02 04:00:21.422869 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.422295 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:21.528303 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.528277 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca\") pod \"294241de-b2e0-4c92-af80-dfdbaa6060c3\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " Nov 02 04:00:21.529094 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.528961 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca" pod "294241de-b2e0-4c92-af80-dfdbaa6060c3" (UID: "294241de-b2e0-4c92-af80-dfdbaa6060c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:21.529517 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.529479 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp\") pod \"294241de-b2e0-4c92-af80-dfdbaa6060c3\" (UID: \"294241de-b2e0-4c92-af80-dfdbaa6060c3\") " Nov 02 04:00:21.530668 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.530642 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/294241de-b2e0-4c92-af80-dfdbaa6060c3-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.563029 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.562997 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp" pod "294241de-b2e0-4c92-af80-dfdbaa6060c3" (UID: "294241de-b2e0-4c92-af80-dfdbaa6060c3"). InnerVolumeSpecName "kube-api-access-7b7gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:21.601324 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.601218 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:21.613673 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.613650 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:21.657374 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.631712 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b7gp\" (UniqueName: \"kubernetes.io/projected/294241de-b2e0-4c92-af80-dfdbaa6060c3-kube-api-access-7b7gp\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.682824 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.682791 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q"] Nov 02 04:00:21.686227 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.686205 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f6e39-29893200-bbr2q"] Nov 02 04:00:21.708418 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.708398 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:21.756878 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.756846 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 02 04:00:21.757042 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.756909 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 02 04:00:21.758147 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758126 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv\" (UniqueName: \"kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv\") pod \"11967c4f-0f40-4152-8e76-2f5648550693\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " Nov 02 04:00:21.758286 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758269 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca\") pod \"09169936-e489-4c22-9d56-8478c2436f68\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " Nov 02 04:00:21.758459 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758442 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca\") pod \"11967c4f-0f40-4152-8e76-2f5648550693\" (UID: \"11967c4f-0f40-4152-8e76-2f5648550693\") " Nov 02 04:00:21.758514 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758489 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw\" (UniqueName: \"kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw\") pod \"09169936-e489-4c22-9d56-8478c2436f68\" (UID: \"09169936-e489-4c22-9d56-8478c2436f68\") " Nov 02 04:00:21.758734 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758707 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca" pod "09169936-e489-4c22-9d56-8478c2436f68" (UID: "09169936-e489-4c22-9d56-8478c2436f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:21.758807 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758772 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca" pod "11967c4f-0f40-4152-8e76-2f5648550693" (UID: "11967c4f-0f40-4152-8e76-2f5648550693"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:21.758975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758959 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09169936-e489-4c22-9d56-8478c2436f68-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.759045 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.758976 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11967c4f-0f40-4152-8e76-2f5648550693-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.760490 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.760466 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv" pod "11967c4f-0f40-4152-8e76-2f5648550693" (UID: "11967c4f-0f40-4152-8e76-2f5648550693"). InnerVolumeSpecName "kube-api-access-2ppxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:21.760580 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.760528 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw" pod "09169936-e489-4c22-9d56-8478c2436f68" (UID: "09169936-e489-4c22-9d56-8478c2436f68"). InnerVolumeSpecName "kube-api-access-r7znw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:21.859492 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.859459 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca\") pod \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " Nov 02 04:00:21.859676 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.859557 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6\" (UniqueName: \"kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6\") pod \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\" (UID: \"ade750b7-8be1-45da-aeb2-cfd2e6cef52b\") " Nov 02 04:00:21.859812 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.859787 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca" pod "ade750b7-8be1-45da-aeb2-cfd2e6cef52b" (UID: "ade750b7-8be1-45da-aeb2-cfd2e6cef52b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:21.860089 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.860065 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-maas-api-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.860226 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.860093 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7znw\" (UniqueName: \"kubernetes.io/projected/09169936-e489-4c22-9d56-8478c2436f68-kube-api-access-r7znw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.860226 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.860108 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppxv\" (UniqueName: \"kubernetes.io/projected/11967c4f-0f40-4152-8e76-2f5648550693-kube-api-access-2ppxv\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:21.861778 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.861755 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6" pod "ade750b7-8be1-45da-aeb2-cfd2e6cef52b" (UID: "ade750b7-8be1-45da-aeb2-cfd2e6cef52b"). InnerVolumeSpecName "kube-api-access-vkdh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:21.961866 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:21.961817 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkdh6\" (UniqueName: \"kubernetes.io/projected/ade750b7-8be1-45da-aeb2-cfd2e6cef52b-kube-api-access-vkdh6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:22.118593 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.118506 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" path="/var/lib/kubelet/pods/09fe7ef7-fb83-47d1-89db-1b9d898bae39/volumes" Nov 02 04:00:22.119088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.119065 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" path="/var/lib/kubelet/pods/4377c8b4-e5d5-4192-a130-0b7385acd12e/volumes" Nov 02 04:00:22.119639 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.119617 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" path="/var/lib/kubelet/pods/a92b6429-925b-420d-b916-fdf9f30e979c/volumes" Nov 02 04:00:22.120389 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.120367 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" path="/var/lib/kubelet/pods/d7258881-8b2d-4a65-8dc6-e59d2d13701c/volumes" Nov 02 04:00:22.120764 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.120746 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" path="/var/lib/kubelet/pods/e85e0a2b-8ce6-4b15-bb89-d017abfb072d/volumes" Nov 02 04:00:22.210666 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.210633 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" Nov 02 04:00:22.210876 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.210631 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4" event={"ID":"ade750b7-8be1-45da-aeb2-cfd2e6cef52b","Type":"ContainerDied","Data":"9c63cdd640948271b8b313080c823456b7746c84a38bb2559958ee0ff134589e"} Nov 02 04:00:22.210876 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.210792 4868 scope.go:122] "RemoveContainer" containerID="28ab160f99d597ccb9033e6dd403fdba496a53b638a83fe61bcbb3c2ff467c83" Nov 02 04:00:22.212449 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.212062 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" Nov 02 04:00:22.212449 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.212087 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds" event={"ID":"11967c4f-0f40-4152-8e76-2f5648550693","Type":"ContainerDied","Data":"e60c42249923a33b33da12b295966fec7191f90840d2b6ad7affe3d6073eb31f"} Nov 02 04:00:22.214257 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.214236 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" Nov 02 04:00:22.214772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.214694 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6" event={"ID":"294241de-b2e0-4c92-af80-dfdbaa6060c3","Type":"ContainerDied","Data":"099b57705721b877cce063e622b3771b580ca568c879d36d3823cf831c1360b6"} Nov 02 04:00:22.216862 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.216835 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" event={"ID":"09169936-e489-4c22-9d56-8478c2436f68","Type":"ContainerDied","Data":"e025c05abe9fed748dfbd5e7d8ca0b4a70a0c9073ac28ecc3b7ed295a06f8aed"} Nov 02 04:00:22.216949 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.216928 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp" Nov 02 04:00:22.224627 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.224604 4868 scope.go:122] "RemoveContainer" containerID="0b87ea7f04377d72f6ab5e5356da900adaa7ca1e0aaae06d4b566f2b74246750" Nov 02 04:00:22.233744 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.233727 4868 scope.go:122] "RemoveContainer" containerID="bb1fff225302174d395ced7b654815ef2658e60016772802c446d803aef10df4" Nov 02 04:00:22.243738 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.243717 4868 scope.go:122] "RemoveContainer" containerID="15191fcf8ad71a8cbe346fa48190d92896c835a3ff4f91a815a1425210c9d303" Nov 02 04:00:22.492742 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.492709 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv"] Nov 02 04:00:22.500513 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.500487 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ac7ab4-29893200-k9rtv"] Nov 02 04:00:22.608422 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.608390 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds"] Nov 02 04:00:22.612219 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.612192 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-398828-29893200-l5zds"] Nov 02 04:00:22.727181 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.727119 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp"] Nov 02 04:00:22.731572 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.731549 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aad578-29893200-gr8pp"] Nov 02 04:00:22.746631 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.746610 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 04:00:22.747565 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.747550 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 04:00:22.838837 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.838808 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4"] Nov 02 04:00:22.842316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.842293 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4c3ea5-29893200-4zvb4"] Nov 02 04:00:22.951805 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.951772 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6"] Nov 02 04:00:22.957149 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:22.957125 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-147104-29893200-bwxh6"] Nov 02 04:00:24.118635 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:24.118603 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09169936-e489-4c22-9d56-8478c2436f68" path="/var/lib/kubelet/pods/09169936-e489-4c22-9d56-8478c2436f68/volumes" Nov 02 04:00:24.119034 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:24.119002 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11967c4f-0f40-4152-8e76-2f5648550693" path="/var/lib/kubelet/pods/11967c4f-0f40-4152-8e76-2f5648550693/volumes" Nov 02 04:00:24.119375 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:24.119346 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" path="/var/lib/kubelet/pods/294241de-b2e0-4c92-af80-dfdbaa6060c3/volumes" Nov 02 04:00:24.119991 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:24.119976 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" path="/var/lib/kubelet/pods/3760203b-bcac-4f6a-a50b-9943eedb32aa/volumes" Nov 02 04:00:24.120337 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:24.120325 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" path="/var/lib/kubelet/pods/ade750b7-8be1-45da-aeb2-cfd2e6cef52b/volumes" Nov 02 04:00:31.622539 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.622505 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt"] Nov 02 04:00:31.623090 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623073 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623090 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623101 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623106 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623115 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623123 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623137 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623142 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623152 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623157 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623163 4868 cpu_manager.go:414] "containerMap: removing container" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623168 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623175 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623179 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623188 4868 cpu_manager.go:414] "containerMap: removing container" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623194 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623203 4868 cpu_manager.go:414] "containerMap: removing container" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623210 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623217 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623221 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623227 4868 cpu_manager.go:414] "containerMap: removing container" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623233 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623242 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623246 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623253 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623257 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623263 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623268 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623277 4868 cpu_manager.go:414] "containerMap: removing container" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623281 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623290 4868 cpu_manager.go:414] "containerMap: removing container" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623295 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623306 4868 cpu_manager.go:414] "containerMap: removing container" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623312 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623318 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623323 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623328 4868 cpu_manager.go:414] "containerMap: removing container" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623332 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623343 4868 cpu_manager.go:414] "containerMap: removing container" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623347 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623366 4868 cpu_manager.go:414] "containerMap: removing container" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623370 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623379 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623383 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623389 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623394 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623401 4868 cpu_manager.go:414] "containerMap: removing container" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623405 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623414 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623418 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623423 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623427 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623436 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623443 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623454 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623459 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623565 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623573 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623579 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623584 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623591 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623596 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623601 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623606 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623610 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623614 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623619 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623625 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623630 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623636 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623641 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623646 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="09169936-e489-4c22-9d56-8478c2436f68" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623654 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623663 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623669 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623676 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623681 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623687 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623692 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623698 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623703 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623708 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e19ea6-be5d-459a-a313-cb6b8d925fd3" containerName="cleanup" Nov 02 04:00:31.623780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.623713 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:31.629127 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.629108 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.656142 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.656118 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.656266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.656245 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.656318 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.656294 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6pw\" (UniqueName: \"kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.683731 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.683706 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt"] Nov 02 04:00:31.728623 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.728596 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:00:31.728746 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.728730 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:00:31.757500 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.757473 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.757604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.757514 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6pw\" (UniqueName: \"kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.757604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.757575 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.758173 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.758155 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.759873 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.759843 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.813672 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.813643 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6pw\" (UniqueName: \"kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw\") pod \"maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:31.943674 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:31.943609 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:32.316865 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:32.316827 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt"] Nov 02 04:00:32.317310 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:32.317284 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de15e57_7a85_4fe6_8947_e5d815036975.slice/crio-28f0375e16534a4edb0ab83fececd6561bbd24d683b2a1fa183027e46d6d4f51 WatchSource:0}: Error finding container 28f0375e16534a4edb0ab83fececd6561bbd24d683b2a1fa183027e46d6d4f51: Status 404 returned error can't find the container with id 28f0375e16534a4edb0ab83fececd6561bbd24d683b2a1fa183027e46d6d4f51 Nov 02 04:00:33.296470 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:33.296435 4868 generic.go:356] "Generic (PLEG): container finished" podID="4de15e57-7a85-4fe6-8947-e5d815036975" containerID="460a7b60b5f8dcb41780450c453f6fb37eaa0c31d84fabde6aebcef03165034d" exitCode=0 Nov 02 04:00:33.296858 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:33.296524 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" event={"ID":"4de15e57-7a85-4fe6-8947-e5d815036975","Type":"ContainerDied","Data":"460a7b60b5f8dcb41780450c453f6fb37eaa0c31d84fabde6aebcef03165034d"} Nov 02 04:00:33.296858 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:33.296578 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" event={"ID":"4de15e57-7a85-4fe6-8947-e5d815036975","Type":"ContainerStarted","Data":"28f0375e16534a4edb0ab83fececd6561bbd24d683b2a1fa183027e46d6d4f51"} Nov 02 04:00:33.445799 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:33.445766 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:00:34.448033 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.448009 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:34.487364 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.487327 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca\") pod \"4de15e57-7a85-4fe6-8947-e5d815036975\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " Nov 02 04:00:34.487579 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.487560 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw\" (UniqueName: \"kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw\") pod \"4de15e57-7a85-4fe6-8947-e5d815036975\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " Nov 02 04:00:34.487622 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.487613 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca\") pod \"4de15e57-7a85-4fe6-8947-e5d815036975\" (UID: \"4de15e57-7a85-4fe6-8947-e5d815036975\") " Nov 02 04:00:34.487668 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.487627 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca" pod "4de15e57-7a85-4fe6-8947-e5d815036975" (UID: "4de15e57-7a85-4fe6-8947-e5d815036975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:34.488073 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.488052 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:34.489579 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.489559 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw" pod "4de15e57-7a85-4fe6-8947-e5d815036975" (UID: "4de15e57-7a85-4fe6-8947-e5d815036975"). InnerVolumeSpecName "kube-api-access-gp6pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:34.489696 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.489679 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca" pod "4de15e57-7a85-4fe6-8947-e5d815036975" (UID: "4de15e57-7a85-4fe6-8947-e5d815036975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:00:34.588925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.588864 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6pw\" (UniqueName: \"kubernetes.io/projected/4de15e57-7a85-4fe6-8947-e5d815036975-kube-api-access-gp6pw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:34.588925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:34.588891 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de15e57-7a85-4fe6-8947-e5d815036975-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:35.310048 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:35.310022 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" Nov 02 04:00:35.310048 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:35.310033 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt" event={"ID":"4de15e57-7a85-4fe6-8947-e5d815036975","Type":"ContainerDied","Data":"28f0375e16534a4edb0ab83fececd6561bbd24d683b2a1fa183027e46d6d4f51"} Nov 02 04:00:35.310273 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:35.310080 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28f0375e16534a4edb0ab83fececd6561bbd24d683b2a1fa183027e46d6d4f51" Nov 02 04:00:39.729030 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.728988 4868 scope.go:122] "RemoveContainer" containerID="b42a92de74b9d59588dde2bdb40fb76020818b74b17b1e881f8441345d43882e" Nov 02 04:00:39.843276 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843236 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q"] Nov 02 04:00:39.843818 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843804 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4de15e57-7a85-4fe6-8947-e5d815036975" containerName="revoke-keys" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843822 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de15e57-7a85-4fe6-8947-e5d815036975" containerName="revoke-keys" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843844 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843850 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade750b7-8be1-45da-aeb2-cfd2e6cef52b" containerName="cleanup" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843859 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843864 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843870 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:39.843875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843874 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843884 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843889 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85e0a2b-8ce6-4b15-bb89-d017abfb072d" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843914 4868 cpu_manager.go:414] "containerMap: removing container" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.843919 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fe7ef7-fb83-47d1-89db-1b9d898bae39" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844005 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de15e57-7a85-4fe6-8947-e5d815036975" containerName="revoke-keys" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844016 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7258881-8b2d-4a65-8dc6-e59d2d13701c" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844023 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92b6429-925b-420d-b916-fdf9f30e979c" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844029 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3760203b-bcac-4f6a-a50b-9943eedb32aa" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844035 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="11967c4f-0f40-4152-8e76-2f5648550693" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844043 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4377c8b4-e5d5-4192-a130-0b7385acd12e" containerName="cleanup" Nov 02 04:00:39.844134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.844047 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="294241de-b2e0-4c92-af80-dfdbaa6060c3" containerName="cleanup" Nov 02 04:00:39.849189 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.849172 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:39.912519 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.912483 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q"] Nov 02 04:00:39.940483 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.940457 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rf2l\" (UniqueName: \"kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:39.940659 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.940640 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:39.940775 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.940759 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:39.957027 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.957002 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:00:39.957027 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:39.957018 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:00:40.041489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.041421 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.041489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.041463 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rf2l\" (UniqueName: \"kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.041689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.041587 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.042198 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.042173 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.043818 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.043796 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.099533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.099504 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rf2l\" (UniqueName: \"kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l\") pod \"maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.161932 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.161904 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:40.541168 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:40.541143 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q"] Nov 02 04:00:40.542358 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:00:40.542331 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e723d6_7549_453d_b599_c0984e609a1a.slice/crio-a4f7f35e04a88f94f3ed7cb14ad7fa4f7afd696f9d865d8531c4af612b65aa6d WatchSource:0}: Error finding container a4f7f35e04a88f94f3ed7cb14ad7fa4f7afd696f9d865d8531c4af612b65aa6d: Status 404 returned error can't find the container with id a4f7f35e04a88f94f3ed7cb14ad7fa4f7afd696f9d865d8531c4af612b65aa6d Nov 02 04:00:41.351558 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:41.351520 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" event={"ID":"66e723d6-7549-453d-b599-c0984e609a1a","Type":"ContainerStarted","Data":"ae8ce07d730d7ae22bc6bce5b947ac849ec77de92d6b4c9edd5cfe6f802ac66f"} Nov 02 04:00:41.351953 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:41.351577 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" event={"ID":"66e723d6-7549-453d-b599-c0984e609a1a","Type":"ContainerStarted","Data":"a4f7f35e04a88f94f3ed7cb14ad7fa4f7afd696f9d865d8531c4af612b65aa6d"} Nov 02 04:00:41.480954 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:41.480908 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" podStartSLOduration=2.4808939199999998 podStartE2EDuration="2.48089392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:00:41.478870959 +0000 UTC m=+4383.474571904" watchObservedRunningTime="2026-11-02 04:00:41.48089392 +0000 UTC m=+4383.476594864" Nov 02 04:00:42.130511 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:42.130475 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g"] Nov 02 04:00:42.236320 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:42.236279 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75"] Nov 02 04:00:42.359638 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:42.359604 4868 generic.go:356] "Generic (PLEG): container finished" podID="66e723d6-7549-453d-b599-c0984e609a1a" containerID="ae8ce07d730d7ae22bc6bce5b947ac849ec77de92d6b4c9edd5cfe6f802ac66f" exitCode=0 Nov 02 04:00:42.360055 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:42.359688 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" event={"ID":"66e723d6-7549-453d-b599-c0984e609a1a","Type":"ContainerDied","Data":"ae8ce07d730d7ae22bc6bce5b947ac849ec77de92d6b4c9edd5cfe6f802ac66f"} Nov 02 04:00:43.511303 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.511281 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:43.584109 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.584079 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l\" (UniqueName: \"kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l\") pod \"66e723d6-7549-453d-b599-c0984e609a1a\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " Nov 02 04:00:43.584266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.584155 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca\") pod \"66e723d6-7549-453d-b599-c0984e609a1a\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " Nov 02 04:00:43.584266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.584209 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca\") pod \"66e723d6-7549-453d-b599-c0984e609a1a\" (UID: \"66e723d6-7549-453d-b599-c0984e609a1a\") " Nov 02 04:00:43.584510 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.584489 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca" pod "66e723d6-7549-453d-b599-c0984e609a1a" (UID: "66e723d6-7549-453d-b599-c0984e609a1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:43.584597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.584578 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:43.586216 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.586192 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca" pod "66e723d6-7549-453d-b599-c0984e609a1a" (UID: "66e723d6-7549-453d-b599-c0984e609a1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:00:43.586324 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.586250 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l" pod "66e723d6-7549-453d-b599-c0984e609a1a" (UID: "66e723d6-7549-453d-b599-c0984e609a1a"). InnerVolumeSpecName "kube-api-access-4rf2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:43.692248 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.692157 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66e723d6-7549-453d-b599-c0984e609a1a-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:43.692492 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:43.692470 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rf2l\" (UniqueName: \"kubernetes.io/projected/66e723d6-7549-453d-b599-c0984e609a1a-kube-api-access-4rf2l\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:44.374875 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:44.374843 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" Nov 02 04:00:44.375054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:44.374881 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q" event={"ID":"66e723d6-7549-453d-b599-c0984e609a1a","Type":"ContainerDied","Data":"a4f7f35e04a88f94f3ed7cb14ad7fa4f7afd696f9d865d8531c4af612b65aa6d"} Nov 02 04:00:44.375054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:44.374961 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4f7f35e04a88f94f3ed7cb14ad7fa4f7afd696f9d865d8531c4af612b65aa6d" Nov 02 04:00:47.131649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.131596 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" podUID="be9df8a5-1612-4991-a62d-f8509a92241a" containerName="payload-processing" containerID="cri-o://7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca" gracePeriod=25 Nov 02 04:00:47.236880 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.236841 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" podUID="7a2e4743-1cd0-4914-a9d5-6310a23a86ce" containerName="payload-pre-processing" containerID="cri-o://d0df4bae75813c50df39a44e9a91cc85e7a768c1f4331c455f92062da2f79758" gracePeriod=25 Nov 02 04:00:47.392895 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.392830 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 04:00:47.404751 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.404717 4868 generic.go:356] "Generic (PLEG): container finished" podID="7a2e4743-1cd0-4914-a9d5-6310a23a86ce" containerID="d0df4bae75813c50df39a44e9a91cc85e7a768c1f4331c455f92062da2f79758" exitCode=0 Nov 02 04:00:47.404873 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.404861 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" event={"ID":"7a2e4743-1cd0-4914-a9d5-6310a23a86ce","Type":"ContainerDied","Data":"d0df4bae75813c50df39a44e9a91cc85e7a768c1f4331c455f92062da2f79758"} Nov 02 04:00:47.406475 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.406453 4868 generic.go:356] "Generic (PLEG): container finished" podID="be9df8a5-1612-4991-a62d-f8509a92241a" containerID="7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca" exitCode=0 Nov 02 04:00:47.406596 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.406506 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" event={"ID":"be9df8a5-1612-4991-a62d-f8509a92241a","Type":"ContainerDied","Data":"7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca"} Nov 02 04:00:47.406596 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.406527 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" Nov 02 04:00:47.406596 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.406545 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g" event={"ID":"be9df8a5-1612-4991-a62d-f8509a92241a","Type":"ContainerDied","Data":"dd7c40ae11648f24e4d7a2498d750ac95e9e9539b0012ec38b1b35790de5877c"} Nov 02 04:00:47.406596 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.406572 4868 scope.go:122] "RemoveContainer" containerID="7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca" Nov 02 04:00:47.418804 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.418781 4868 scope.go:122] "RemoveContainer" containerID="7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca" Nov 02 04:00:47.419223 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:00:47.419193 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca\": container with ID starting with 7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca not found: ID does not exist" containerID="7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca" Nov 02 04:00:47.419620 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.419233 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca"} err="failed to get container status \"7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca\": rpc error: code = NotFound desc = could not find container \"7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca\": container with ID starting with 7e9419238b29262983c89509b14005fd66d85d7882e21101aba4c42976ce17ca not found: ID does not exist" Nov 02 04:00:47.437570 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.437546 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume\") pod \"be9df8a5-1612-4991-a62d-f8509a92241a\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " Nov 02 04:00:47.437669 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.437584 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc\" (UniqueName: \"kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc\") pod \"be9df8a5-1612-4991-a62d-f8509a92241a\" (UID: \"be9df8a5-1612-4991-a62d-f8509a92241a\") " Nov 02 04:00:47.437939 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.437916 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume" pod "be9df8a5-1612-4991-a62d-f8509a92241a" (UID: "be9df8a5-1612-4991-a62d-f8509a92241a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:47.441225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.441198 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc" pod "be9df8a5-1612-4991-a62d-f8509a92241a" (UID: "be9df8a5-1612-4991-a62d-f8509a92241a"). InnerVolumeSpecName "kube-api-access-vhkkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:47.499592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.499571 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 04:00:47.538489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.538445 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume\") pod \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " Nov 02 04:00:47.538658 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.538638 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb\" (UniqueName: \"kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb\") pod \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\" (UID: \"7a2e4743-1cd0-4914-a9d5-6310a23a86ce\") " Nov 02 04:00:47.538889 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.538861 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume" pod "7a2e4743-1cd0-4914-a9d5-6310a23a86ce" (UID: "7a2e4743-1cd0-4914-a9d5-6310a23a86ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:47.539178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.539157 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be9df8a5-1612-4991-a62d-f8509a92241a-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:47.539230 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.539186 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhkkc\" (UniqueName: \"kubernetes.io/projected/be9df8a5-1612-4991-a62d-f8509a92241a-kube-api-access-vhkkc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:47.539230 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.539203 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:47.540608 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.540586 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb" pod "7a2e4743-1cd0-4914-a9d5-6310a23a86ce" (UID: "7a2e4743-1cd0-4914-a9d5-6310a23a86ce"). InnerVolumeSpecName "kube-api-access-n6stb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:47.640774 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.640744 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6stb\" (UniqueName: \"kubernetes.io/projected/7a2e4743-1cd0-4914-a9d5-6310a23a86ce-kube-api-access-n6stb\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:47.833865 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.833822 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g"] Nov 02 04:00:47.836787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:47.836760 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-61c297-6df8d4c95f-rkt4g"] Nov 02 04:00:48.115465 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:48.115438 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9df8a5-1612-4991-a62d-f8509a92241a" path="/var/lib/kubelet/pods/be9df8a5-1612-4991-a62d-f8509a92241a/volumes" Nov 02 04:00:48.414021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:48.413938 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" Nov 02 04:00:48.414021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:48.413965 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75" event={"ID":"7a2e4743-1cd0-4914-a9d5-6310a23a86ce","Type":"ContainerDied","Data":"2094f0ccf1cd7627d7ff6743afe4690eb47895c55801a7ac2372a0f4f3832f42"} Nov 02 04:00:48.414628 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:48.414032 4868 scope.go:122] "RemoveContainer" containerID="d0df4bae75813c50df39a44e9a91cc85e7a768c1f4331c455f92062da2f79758" Nov 02 04:00:48.580152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:48.580121 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75"] Nov 02 04:00:48.588300 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:48.588270 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61c297-ddc8b5c4c-j7k75"] Nov 02 04:00:50.114820 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:50.114786 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2e4743-1cd0-4914-a9d5-6310a23a86ce" path="/var/lib/kubelet/pods/7a2e4743-1cd0-4914-a9d5-6310a23a86ce/volumes" Nov 02 04:00:52.250575 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:52.250543 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9"] Nov 02 04:00:57.251509 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.251468 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" podUID="2ddd750e-10d3-4dc4-a582-308c1e79c3e6" containerName="payload-pre-processing" containerID="cri-o://fd8f7d8a9d6c3e63fec6ad0a3beb94b3f35624ad374260f1ff4a4351aa31bb28" gracePeriod=25 Nov 02 04:00:57.474380 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.474327 4868 generic.go:356] "Generic (PLEG): container finished" podID="2ddd750e-10d3-4dc4-a582-308c1e79c3e6" containerID="fd8f7d8a9d6c3e63fec6ad0a3beb94b3f35624ad374260f1ff4a4351aa31bb28" exitCode=0 Nov 02 04:00:57.474559 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.474397 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" event={"ID":"2ddd750e-10d3-4dc4-a582-308c1e79c3e6","Type":"ContainerDied","Data":"fd8f7d8a9d6c3e63fec6ad0a3beb94b3f35624ad374260f1ff4a4351aa31bb28"} Nov 02 04:00:57.531163 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.531144 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 04:00:57.651136 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.651108 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume\") pod \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " Nov 02 04:00:57.651327 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.651147 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w\" (UniqueName: \"kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w\") pod \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\" (UID: \"2ddd750e-10d3-4dc4-a582-308c1e79c3e6\") " Nov 02 04:00:57.651579 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.651555 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume" pod "2ddd750e-10d3-4dc4-a582-308c1e79c3e6" (UID: "2ddd750e-10d3-4dc4-a582-308c1e79c3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:00:57.651678 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.651664 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:57.653382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.653335 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w" pod "2ddd750e-10d3-4dc4-a582-308c1e79c3e6" (UID: "2ddd750e-10d3-4dc4-a582-308c1e79c3e6"). InnerVolumeSpecName "kube-api-access-w998w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:00:57.752648 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:57.752619 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w998w\" (UniqueName: \"kubernetes.io/projected/2ddd750e-10d3-4dc4-a582-308c1e79c3e6-kube-api-access-w998w\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:00:58.481610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:58.481526 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" Nov 02 04:00:58.481610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:58.481562 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9" event={"ID":"2ddd750e-10d3-4dc4-a582-308c1e79c3e6","Type":"ContainerDied","Data":"83bc0a731eb00f3448f1f8ebc1b981afea56f3aef0945e4a746b111bc34845ab"} Nov 02 04:00:58.482118 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:58.481632 4868 scope.go:122] "RemoveContainer" containerID="fd8f7d8a9d6c3e63fec6ad0a3beb94b3f35624ad374260f1ff4a4351aa31bb28" Nov 02 04:00:58.655031 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:58.654992 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9"] Nov 02 04:00:58.658156 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:00:58.658126 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9b7f43-665cf654bc-g6jp9"] Nov 02 04:01:00.114579 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:00.114547 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddd750e-10d3-4dc4-a582-308c1e79c3e6" path="/var/lib/kubelet/pods/2ddd750e-10d3-4dc4-a582-308c1e79c3e6/volumes" Nov 02 04:01:03.446227 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:01:03.446187 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:01:09.800817 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.800786 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc"] Nov 02 04:01:09.801297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801283 4868 cpu_manager.go:414] "containerMap: removing container" podUID="66e723d6-7549-453d-b599-c0984e609a1a" containerName="revoke-keys" Nov 02 04:01:09.801339 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801297 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e723d6-7549-453d-b599-c0984e609a1a" containerName="revoke-keys" Nov 02 04:01:09.801339 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801311 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7a2e4743-1cd0-4914-a9d5-6310a23a86ce" containerName="payload-pre-processing" Nov 02 04:01:09.801339 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801317 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2e4743-1cd0-4914-a9d5-6310a23a86ce" containerName="payload-pre-processing" Nov 02 04:01:09.801339 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801333 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2ddd750e-10d3-4dc4-a582-308c1e79c3e6" containerName="payload-pre-processing" Nov 02 04:01:09.801339 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801338 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddd750e-10d3-4dc4-a582-308c1e79c3e6" containerName="payload-pre-processing" Nov 02 04:01:09.801533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801367 4868 cpu_manager.go:414] "containerMap: removing container" podUID="be9df8a5-1612-4991-a62d-f8509a92241a" containerName="payload-processing" Nov 02 04:01:09.801533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801373 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9df8a5-1612-4991-a62d-f8509a92241a" containerName="payload-processing" Nov 02 04:01:09.801533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801462 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e723d6-7549-453d-b599-c0984e609a1a" containerName="revoke-keys" Nov 02 04:01:09.801533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801473 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2e4743-1cd0-4914-a9d5-6310a23a86ce" containerName="payload-pre-processing" Nov 02 04:01:09.801533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801480 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddd750e-10d3-4dc4-a582-308c1e79c3e6" containerName="payload-pre-processing" Nov 02 04:01:09.801533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.801487 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9df8a5-1612-4991-a62d-f8509a92241a" containerName="payload-processing" Nov 02 04:01:09.806063 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.806046 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.858425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.858395 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:01:09.861702 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.861678 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc"] Nov 02 04:01:09.879097 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.879069 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.879200 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.879157 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.879387 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.879340 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vr8\" (UniqueName: \"kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.907012 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.906959 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:01:09.980185 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.980146 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vr8\" (UniqueName: \"kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.980328 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.980203 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.980328 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.980314 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.980899 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.980872 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:09.982813 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:09.982786 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:10.036772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:10.036740 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vr8\" (UniqueName: \"kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:10.118690 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:10.118658 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:10.501415 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:10.501388 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc"] Nov 02 04:01:10.501867 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:01:10.501841 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21029201_19f2_4c38_85c6_997809d0fd71.slice/crio-a1fc609ea4aaa566deb8491fbd45b64718523780aee115ae08716ba11386b423 WatchSource:0}: Error finding container a1fc609ea4aaa566deb8491fbd45b64718523780aee115ae08716ba11386b423: Status 404 returned error can't find the container with id a1fc609ea4aaa566deb8491fbd45b64718523780aee115ae08716ba11386b423 Nov 02 04:01:10.563507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:10.563476 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" event={"ID":"21029201-19f2-4c38-85c6-997809d0fd71","Type":"ContainerStarted","Data":"a1fc609ea4aaa566deb8491fbd45b64718523780aee115ae08716ba11386b423"} Nov 02 04:01:11.570960 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:11.570930 4868 generic.go:356] "Generic (PLEG): container finished" podID="21029201-19f2-4c38-85c6-997809d0fd71" containerID="94078eb0ce79fdc734d242c42ad6d5d08883f5ba03ba2c9d151d328bb73d2c71" exitCode=0 Nov 02 04:01:11.571310 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:11.570980 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" event={"ID":"21029201-19f2-4c38-85c6-997809d0fd71","Type":"ContainerDied","Data":"94078eb0ce79fdc734d242c42ad6d5d08883f5ba03ba2c9d151d328bb73d2c71"} Nov 02 04:01:12.720908 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.720882 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:12.811571 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.811548 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8\" (UniqueName: \"kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8\") pod \"21029201-19f2-4c38-85c6-997809d0fd71\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " Nov 02 04:01:12.811688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.811641 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca\") pod \"21029201-19f2-4c38-85c6-997809d0fd71\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " Nov 02 04:01:12.811688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.811677 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca\") pod \"21029201-19f2-4c38-85c6-997809d0fd71\" (UID: \"21029201-19f2-4c38-85c6-997809d0fd71\") " Nov 02 04:01:12.812073 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.812040 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca" pod "21029201-19f2-4c38-85c6-997809d0fd71" (UID: "21029201-19f2-4c38-85c6-997809d0fd71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:01:12.813653 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.813629 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8" pod "21029201-19f2-4c38-85c6-997809d0fd71" (UID: "21029201-19f2-4c38-85c6-997809d0fd71"). InnerVolumeSpecName "kube-api-access-k6vr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:01:12.813768 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.813656 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca" pod "21029201-19f2-4c38-85c6-997809d0fd71" (UID: "21029201-19f2-4c38-85c6-997809d0fd71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:01:12.913376 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.913326 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vr8\" (UniqueName: \"kubernetes.io/projected/21029201-19f2-4c38-85c6-997809d0fd71-kube-api-access-k6vr8\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:12.913376 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.913376 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:12.913542 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:12.913389 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21029201-19f2-4c38-85c6-997809d0fd71-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:13.584865 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:13.584835 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" Nov 02 04:01:13.585057 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:13.584867 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc" event={"ID":"21029201-19f2-4c38-85c6-997809d0fd71","Type":"ContainerDied","Data":"a1fc609ea4aaa566deb8491fbd45b64718523780aee115ae08716ba11386b423"} Nov 02 04:01:13.585057 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:13.584915 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1fc609ea4aaa566deb8491fbd45b64718523780aee115ae08716ba11386b423" Nov 02 04:01:21.286830 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:21.286794 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw"] Nov 02 04:01:26.287317 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.287253 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" podUID="3c8f7864-786f-4f78-86ab-63769cb63896" containerName="payload-processing" containerID="cri-o://317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356" gracePeriod=25 Nov 02 04:01:26.598400 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.598371 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 04:01:26.670134 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.670104 4868 generic.go:356] "Generic (PLEG): container finished" podID="3c8f7864-786f-4f78-86ab-63769cb63896" containerID="317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356" exitCode=0 Nov 02 04:01:26.670322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.670150 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" Nov 02 04:01:26.670322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.670185 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" event={"ID":"3c8f7864-786f-4f78-86ab-63769cb63896","Type":"ContainerDied","Data":"317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356"} Nov 02 04:01:26.670322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.670250 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw" event={"ID":"3c8f7864-786f-4f78-86ab-63769cb63896","Type":"ContainerDied","Data":"69567e600e414ea590670b14857220ead4ea1b1ec1cfdecc85a5b2d1f8902742"} Nov 02 04:01:26.670322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.670281 4868 scope.go:122] "RemoveContainer" containerID="317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356" Nov 02 04:01:26.682804 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.682785 4868 scope.go:122] "RemoveContainer" containerID="317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356" Nov 02 04:01:26.683208 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:01:26.683181 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356\": container with ID starting with 317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356 not found: ID does not exist" containerID="317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356" Nov 02 04:01:26.683309 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.683216 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356"} err="failed to get container status \"317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356\": rpc error: code = NotFound desc = could not find container \"317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356\": container with ID starting with 317b20af8b0a8becc63a76ca65a5599a42c2bb3cb161584447ada08834e40356 not found: ID does not exist" Nov 02 04:01:26.684630 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.684570 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume\") pod \"3c8f7864-786f-4f78-86ab-63769cb63896\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " Nov 02 04:01:26.684779 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.684755 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs\" (UniqueName: \"kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs\") pod \"3c8f7864-786f-4f78-86ab-63769cb63896\" (UID: \"3c8f7864-786f-4f78-86ab-63769cb63896\") " Nov 02 04:01:26.684950 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.684931 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume" pod "3c8f7864-786f-4f78-86ab-63769cb63896" (UID: "3c8f7864-786f-4f78-86ab-63769cb63896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:01:26.685166 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.685150 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8f7864-786f-4f78-86ab-63769cb63896-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:26.687230 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.687199 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs" pod "3c8f7864-786f-4f78-86ab-63769cb63896" (UID: "3c8f7864-786f-4f78-86ab-63769cb63896"). InnerVolumeSpecName "kube-api-access-7cccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:01:26.786641 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:26.786608 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cccs\" (UniqueName: \"kubernetes.io/projected/3c8f7864-786f-4f78-86ab-63769cb63896-kube-api-access-7cccs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:27.090249 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:27.090213 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw"] Nov 02 04:01:27.161262 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:27.161226 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae1315-864c8985d7-t58sw"] Nov 02 04:01:28.114933 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:28.114901 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8f7864-786f-4f78-86ab-63769cb63896" path="/var/lib/kubelet/pods/3c8f7864-786f-4f78-86ab-63769cb63896/volumes" Nov 02 04:01:30.057770 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:30.057733 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b"] Nov 02 04:01:30.097662 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:30.097633 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-087fcdb0-v495b"] Nov 02 04:01:30.116161 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:30.116133 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327a2ea6-62db-450c-8926-d9371bd0e97d" path="/var/lib/kubelet/pods/327a2ea6-62db-450c-8926-d9371bd0e97d/volumes" Nov 02 04:01:33.447580 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:01:33.447537 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:01:34.726848 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:34.726813 4868 generic.go:356] "Generic (PLEG): container finished" podID="01774e87-b148-4801-b4a8-7bf0564cd846" containerID="bb60c3eb662b9c8789a7f70d10296c68079ca715b87193c358919c464cf6fc29" exitCode=0 Nov 02 04:01:34.727266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:34.726886 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" event={"ID":"01774e87-b148-4801-b4a8-7bf0564cd846","Type":"ContainerDied","Data":"bb60c3eb662b9c8789a7f70d10296c68079ca715b87193c358919c464cf6fc29"} Nov 02 04:01:34.727266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:34.727236 4868 scope.go:122] "RemoveContainer" containerID="bb60c3eb662b9c8789a7f70d10296c68079ca715b87193c358919c464cf6fc29" Nov 02 04:01:34.728698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:34.728670 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:01:34.728773 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:34.728740 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:01:34.728811 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:34.728771 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:01:35.760485 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:35.760449 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" event={"ID":"01774e87-b148-4801-b4a8-7bf0564cd846","Type":"ContainerStarted","Data":"2b2c5be07ffd457513f818fd424447e1c686266bb9c52c52b12aa9a03b2c20f7"} Nov 02 04:01:36.390598 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:36.390558 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 04:01:36.749431 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:36.735619 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 04:01:36.756783 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:36.756457 4868 patch_prober.go:29] interesting pod/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 04:01:36.756783 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:36.756528 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 04:01:37.737386 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:37.735655 4868 patch_prober.go:29] interesting pod/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 04:01:37.737386 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:37.735717 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 04:01:38.750480 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:38.750439 4868 patch_prober.go:29] interesting pod/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 04:01:38.751101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:38.751065 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 04:01:39.736485 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:39.736448 4868 patch_prober.go:29] interesting pod/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 04:01:39.736801 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:39.736772 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 04:01:40.060202 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:40.060116 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg"] Nov 02 04:01:40.114767 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:40.114739 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efd0c24-jg2kg"] Nov 02 04:01:40.731271 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:40.731235 4868 patch_prober.go:29] interesting pod/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 04:01:40.731493 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:40.731290 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 04:01:41.736129 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:41.736102 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 04:01:41.737072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:41.737052 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 04:01:42.118100 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:42.118066 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6" path="/var/lib/kubelet/pods/d15ff5ed-859b-4e3a-b1b4-89af12ff8ac6/volumes" Nov 02 04:01:46.005238 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005205 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg"] Nov 02 04:01:46.005728 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005707 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3c8f7864-786f-4f78-86ab-63769cb63896" containerName="payload-processing" Nov 02 04:01:46.005728 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005728 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8f7864-786f-4f78-86ab-63769cb63896" containerName="payload-processing" Nov 02 04:01:46.005809 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005750 4868 cpu_manager.go:414] "containerMap: removing container" podUID="21029201-19f2-4c38-85c6-997809d0fd71" containerName="revoke-keys" Nov 02 04:01:46.005809 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005755 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21029201-19f2-4c38-85c6-997809d0fd71" containerName="revoke-keys" Nov 02 04:01:46.005882 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005852 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8f7864-786f-4f78-86ab-63769cb63896" containerName="payload-processing" Nov 02 04:01:46.005882 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.005864 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="21029201-19f2-4c38-85c6-997809d0fd71" containerName="revoke-keys" Nov 02 04:01:46.008933 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.008915 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.026426 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.026403 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.026537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.026470 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q889b\" (UniqueName: \"kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.026631 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.026610 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.065161 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.065136 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg"] Nov 02 04:01:46.110814 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.110788 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:01:46.110957 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.110855 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:01:46.128139 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.128105 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.128266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.128210 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.128661 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.128332 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q889b\" (UniqueName: \"kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.128909 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.128893 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.130786 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.130765 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.184238 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.184219 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q889b\" (UniqueName: \"kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b\") pod \"maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.321216 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.321152 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:46.691729 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.691702 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg"] Nov 02 04:01:46.693381 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:01:46.693321 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2756e5_3d00_4a27_bf63_e71f13714b64.slice/crio-43bec77b1c04d0c54529991a529313edcd8920ae8ebcbd305a83238ce6efde38 WatchSource:0}: Error finding container 43bec77b1c04d0c54529991a529313edcd8920ae8ebcbd305a83238ce6efde38: Status 404 returned error can't find the container with id 43bec77b1c04d0c54529991a529313edcd8920ae8ebcbd305a83238ce6efde38 Nov 02 04:01:46.832173 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:46.832138 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" event={"ID":"2e2756e5-3d00-4a27-bf63-e71f13714b64","Type":"ContainerStarted","Data":"43bec77b1c04d0c54529991a529313edcd8920ae8ebcbd305a83238ce6efde38"} Nov 02 04:01:47.848489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:47.848396 4868 generic.go:356] "Generic (PLEG): container finished" podID="2e2756e5-3d00-4a27-bf63-e71f13714b64" containerID="10f87c58ed07108ac452f20255105f14822a25ce6eca1ccced0b6f706b2e1bdb" exitCode=0 Nov 02 04:01:47.848489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:47.848479 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" event={"ID":"2e2756e5-3d00-4a27-bf63-e71f13714b64","Type":"ContainerDied","Data":"10f87c58ed07108ac452f20255105f14822a25ce6eca1ccced0b6f706b2e1bdb"} Nov 02 04:01:49.006684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.006659 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:49.060284 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.060259 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b\" (UniqueName: \"kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b\") pod \"2e2756e5-3d00-4a27-bf63-e71f13714b64\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " Nov 02 04:01:49.060461 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.060405 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca\") pod \"2e2756e5-3d00-4a27-bf63-e71f13714b64\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " Nov 02 04:01:49.060461 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.060438 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca\") pod \"2e2756e5-3d00-4a27-bf63-e71f13714b64\" (UID: \"2e2756e5-3d00-4a27-bf63-e71f13714b64\") " Nov 02 04:01:49.060873 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.060835 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca" pod "2e2756e5-3d00-4a27-bf63-e71f13714b64" (UID: "2e2756e5-3d00-4a27-bf63-e71f13714b64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:01:49.062399 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.062374 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b" pod "2e2756e5-3d00-4a27-bf63-e71f13714b64" (UID: "2e2756e5-3d00-4a27-bf63-e71f13714b64"). InnerVolumeSpecName "kube-api-access-q889b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:01:49.062527 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.062403 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca" pod "2e2756e5-3d00-4a27-bf63-e71f13714b64" (UID: "2e2756e5-3d00-4a27-bf63-e71f13714b64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:01:49.161840 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.161749 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:49.161840 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.161784 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2756e5-3d00-4a27-bf63-e71f13714b64-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:49.161840 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.161796 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q889b\" (UniqueName: \"kubernetes.io/projected/2e2756e5-3d00-4a27-bf63-e71f13714b64-kube-api-access-q889b\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:01:49.863064 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.863036 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" Nov 02 04:01:49.863228 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.863080 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg" event={"ID":"2e2756e5-3d00-4a27-bf63-e71f13714b64","Type":"ContainerDied","Data":"43bec77b1c04d0c54529991a529313edcd8920ae8ebcbd305a83238ce6efde38"} Nov 02 04:01:49.863228 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:49.863127 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bec77b1c04d0c54529991a529313edcd8920ae8ebcbd305a83238ce6efde38" Nov 02 04:01:57.097165 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:57.097133 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k"] Nov 02 04:01:57.146626 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:57.146595 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07c6c42-t7f6k"] Nov 02 04:01:57.606283 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:57.606220 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk"] Nov 02 04:01:58.115524 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:01:58.115493 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648ff475-09ed-4495-a58c-87914523280b" path="/var/lib/kubelet/pods/648ff475-09ed-4495-a58c-87914523280b/volumes" Nov 02 04:02:02.606826 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.606786 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" podUID="9ae2a81c-f24b-46c9-88c3-0721ef2954d6" containerName="payload-pre-processing" containerID="cri-o://3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f" gracePeriod=25 Nov 02 04:02:02.873650 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.873627 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 04:02:02.974508 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.974481 4868 generic.go:356] "Generic (PLEG): container finished" podID="9ae2a81c-f24b-46c9-88c3-0721ef2954d6" containerID="3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f" exitCode=0 Nov 02 04:02:02.974689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.974528 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" Nov 02 04:02:02.974689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.974568 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" event={"ID":"9ae2a81c-f24b-46c9-88c3-0721ef2954d6","Type":"ContainerDied","Data":"3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f"} Nov 02 04:02:02.974689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.974632 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk" event={"ID":"9ae2a81c-f24b-46c9-88c3-0721ef2954d6","Type":"ContainerDied","Data":"5547a199da62f3a1e2b8afa4f192270e40cb172b4bddf70f9bfe0480addf10b0"} Nov 02 04:02:02.974689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.974661 4868 scope.go:122] "RemoveContainer" containerID="3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f" Nov 02 04:02:02.985570 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.985553 4868 scope.go:122] "RemoveContainer" containerID="3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f" Nov 02 04:02:02.985847 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:02.985828 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f\": container with ID starting with 3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f not found: ID does not exist" containerID="3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f" Nov 02 04:02:02.985900 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:02.985857 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f"} err="failed to get container status \"3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f\": rpc error: code = NotFound desc = could not find container \"3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f\": container with ID starting with 3e2f414576546ae05d94bb3751fc1feb291352f9acb57312160298f51774d33f not found: ID does not exist" Nov 02 04:02:03.011568 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.011541 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume\") pod \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " Nov 02 04:02:03.011713 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.011670 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd\" (UniqueName: \"kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd\") pod \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\" (UID: \"9ae2a81c-f24b-46c9-88c3-0721ef2954d6\") " Nov 02 04:02:03.011990 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.011965 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume" pod "9ae2a81c-f24b-46c9-88c3-0721ef2954d6" (UID: "9ae2a81c-f24b-46c9-88c3-0721ef2954d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:03.012107 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.012088 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:03.013749 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.013729 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd" pod "9ae2a81c-f24b-46c9-88c3-0721ef2954d6" (UID: "9ae2a81c-f24b-46c9-88c3-0721ef2954d6"). InnerVolumeSpecName "kube-api-access-xkvrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:03.112942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.112914 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvrd\" (UniqueName: \"kubernetes.io/projected/9ae2a81c-f24b-46c9-88c3-0721ef2954d6-kube-api-access-xkvrd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:03.394771 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.394704 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk"] Nov 02 04:02:03.445103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:03.445075 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-61f3b6-67bb8c8755-9t2pk"] Nov 02 04:02:03.446986 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:03.446956 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:02:04.115131 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:04.115097 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae2a81c-f24b-46c9-88c3-0721ef2954d6" path="/var/lib/kubelet/pods/9ae2a81c-f24b-46c9-88c3-0721ef2954d6/volumes" Nov 02 04:02:07.141982 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:07.141945 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt"] Nov 02 04:02:07.191609 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:07.191584 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61c297-cd4pt"] Nov 02 04:02:08.046073 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:08.046037 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5"] Nov 02 04:02:08.095372 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:08.095327 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96225e55-5ggm5"] Nov 02 04:02:08.115786 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:08.115759 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de15e57-7a85-4fe6-8947-e5d815036975" path="/var/lib/kubelet/pods/4de15e57-7a85-4fe6-8947-e5d815036975/volumes" Nov 02 04:02:08.116195 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:08.116181 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2aaf09-4a9f-4e51-999e-2bb066f7190f" path="/var/lib/kubelet/pods/7a2aaf09-4a9f-4e51-999e-2bb066f7190f/volumes" Nov 02 04:02:13.422898 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:13.422859 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q"] Nov 02 04:02:13.472033 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:13.472002 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9b7f43-cjz4q"] Nov 02 04:02:13.833033 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:13.832953 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp"] Nov 02 04:02:13.833249 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:13.833221 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" containerID="cri-o://563cdb1a4a9b9689f5062de8412c305ec5374d0b5238ab9db73d45d90c548d29" gracePeriod=30 Nov 02 04:02:14.117002 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:14.116973 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e723d6-7549-453d-b599-c0984e609a1a" path="/var/lib/kubelet/pods/66e723d6-7549-453d-b599-c0984e609a1a/volumes" Nov 02 04:02:16.025085 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025051 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5"] Nov 02 04:02:16.025577 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025562 4868 cpu_manager.go:414] "containerMap: removing container" podUID="2e2756e5-3d00-4a27-bf63-e71f13714b64" containerName="revoke-keys" Nov 02 04:02:16.025577 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025577 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2756e5-3d00-4a27-bf63-e71f13714b64" containerName="revoke-keys" Nov 02 04:02:16.025664 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025610 4868 cpu_manager.go:414] "containerMap: removing container" podUID="9ae2a81c-f24b-46c9-88c3-0721ef2954d6" containerName="payload-pre-processing" Nov 02 04:02:16.025664 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025616 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae2a81c-f24b-46c9-88c3-0721ef2954d6" containerName="payload-pre-processing" Nov 02 04:02:16.025749 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025699 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2756e5-3d00-4a27-bf63-e71f13714b64" containerName="revoke-keys" Nov 02 04:02:16.025749 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.025712 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae2a81c-f24b-46c9-88c3-0721ef2954d6" containerName="payload-pre-processing" Nov 02 04:02:16.028614 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.028593 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.085106 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.085079 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5"] Nov 02 04:02:16.131089 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.131060 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:02:16.131254 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.131112 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:02:16.167042 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.167014 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.167196 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.167080 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.167276 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.167255 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkmc\" (UniqueName: \"kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.268834 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.268803 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.269025 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.268905 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkmc\" (UniqueName: \"kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.269025 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.268942 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.269546 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.269521 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.271268 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.271247 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.325769 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.325699 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkmc\" (UniqueName: \"kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.340207 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.340185 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:16.711962 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:16.711933 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5"] Nov 02 04:02:16.712794 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:02:16.712767 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8a330b_c407_496e_ba6b_1a3c14f9366f.slice/crio-1ee0608ee9916a443c1aa4ebaa7aab850a6c1345c33edb8d7ceaf2f08e9aa734 WatchSource:0}: Error finding container 1ee0608ee9916a443c1aa4ebaa7aab850a6c1345c33edb8d7ceaf2f08e9aa734: Status 404 returned error can't find the container with id 1ee0608ee9916a443c1aa4ebaa7aab850a6c1345c33edb8d7ceaf2f08e9aa734 Nov 02 04:02:17.064432 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:17.064405 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" event={"ID":"7d8a330b-c407-496e-ba6b-1a3c14f9366f","Type":"ContainerStarted","Data":"411e1c50d6a7f6a0ca17cc72deb0ed88c9a95a0608d94d56744d880cb09c30d6"} Nov 02 04:02:17.064741 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:17.064455 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" event={"ID":"7d8a330b-c407-496e-ba6b-1a3c14f9366f","Type":"ContainerStarted","Data":"1ee0608ee9916a443c1aa4ebaa7aab850a6c1345c33edb8d7ceaf2f08e9aa734"} Nov 02 04:02:17.181067 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:17.181015 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" podStartSLOduration=2.181002228 podStartE2EDuration="2.181002228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:02:17.177199463 +0000 UTC m=+4479.172900409" watchObservedRunningTime="2026-11-02 04:02:17.181002228 +0000 UTC m=+4479.176703171" Nov 02 04:02:18.072427 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:18.072398 4868 generic.go:356] "Generic (PLEG): container finished" podID="7d8a330b-c407-496e-ba6b-1a3c14f9366f" containerID="411e1c50d6a7f6a0ca17cc72deb0ed88c9a95a0608d94d56744d880cb09c30d6" exitCode=0 Nov 02 04:02:18.072797 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:18.072488 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" event={"ID":"7d8a330b-c407-496e-ba6b-1a3c14f9366f","Type":"ContainerDied","Data":"411e1c50d6a7f6a0ca17cc72deb0ed88c9a95a0608d94d56744d880cb09c30d6"} Nov 02 04:02:19.081080 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.081037 4868 generic.go:356] "Generic (PLEG): container finished" podID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerID="563cdb1a4a9b9689f5062de8412c305ec5374d0b5238ab9db73d45d90c548d29" exitCode=0 Nov 02 04:02:19.081541 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.081105 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" event={"ID":"80ae4893-94e4-484c-bac3-c1f582c7d863","Type":"ContainerDied","Data":"563cdb1a4a9b9689f5062de8412c305ec5374d0b5238ab9db73d45d90c548d29"} Nov 02 04:02:19.081541 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.081184 4868 scope.go:122] "RemoveContainer" containerID="bc0bbc40c7a039e929b15414f149ce93775b64368895634f041d1ef42d7637e0" Nov 02 04:02:19.109922 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.109901 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 04:02:19.207918 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.207885 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208068 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.207951 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208068 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208014 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208161 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208073 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208209 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208166 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208263 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208236 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208327 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208251 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:19.208327 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208289 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208327 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208298 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:19.208327 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208313 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208605 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208392 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data\") pod \"80ae4893-94e4-484c-bac3-c1f582c7d863\" (UID: \"80ae4893-94e4-484c-bac3-c1f582c7d863\") " Nov 02 04:02:19.208737 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208630 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:19.208866 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208840 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:19.208985 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208946 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:19.209062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.208985 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.209062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.209010 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ae4893-94e4-484c-bac3-c1f582c7d863-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.209062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.209025 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.210606 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.210579 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:19.210737 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.210640 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:19.210737 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.210649 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:02:19.211486 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.211459 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr" pod "80ae4893-94e4-484c-bac3-c1f582c7d863" (UID: "80ae4893-94e4-484c-bac3-c1f582c7d863"). InnerVolumeSpecName "kube-api-access-nvtdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:19.239402 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.239378 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:19.310248 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.310217 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.310248 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.310243 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.310248 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.310252 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.310550 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.310260 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.310550 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.310269 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtdr\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-kube-api-access-nvtdr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.310550 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.310279 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ae4893-94e4-484c-bac3-c1f582c7d863-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.410818 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.410759 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc\" (UniqueName: \"kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc\") pod \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " Nov 02 04:02:19.410926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.410858 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca\") pod \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " Nov 02 04:02:19.411001 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.410981 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca\") pod \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\" (UID: \"7d8a330b-c407-496e-ba6b-1a3c14f9366f\") " Nov 02 04:02:19.411262 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.411238 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca" pod "7d8a330b-c407-496e-ba6b-1a3c14f9366f" (UID: "7d8a330b-c407-496e-ba6b-1a3c14f9366f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:19.411413 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.411398 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.412918 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.412896 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca" pod "7d8a330b-c407-496e-ba6b-1a3c14f9366f" (UID: "7d8a330b-c407-496e-ba6b-1a3c14f9366f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:02:19.413006 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.412934 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc" pod "7d8a330b-c407-496e-ba6b-1a3c14f9366f" (UID: "7d8a330b-c407-496e-ba6b-1a3c14f9366f"). InnerVolumeSpecName "kube-api-access-xlkmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:19.512634 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.512604 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8a330b-c407-496e-ba6b-1a3c14f9366f-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:19.512634 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:19.512633 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlkmc\" (UniqueName: \"kubernetes.io/projected/7d8a330b-c407-496e-ba6b-1a3c14f9366f-kube-api-access-xlkmc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:20.088658 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.088629 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" Nov 02 04:02:20.089188 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.088668 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp" event={"ID":"80ae4893-94e4-484c-bac3-c1f582c7d863","Type":"ContainerDied","Data":"5084c169f859cdbdaaeb215e44535370defb5dc605c34717fb1b3682fa96d921"} Nov 02 04:02:20.089188 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.088725 4868 scope.go:122] "RemoveContainer" containerID="563cdb1a4a9b9689f5062de8412c305ec5374d0b5238ab9db73d45d90c548d29" Nov 02 04:02:20.090181 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.090165 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" Nov 02 04:02:20.090282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.090186 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5" event={"ID":"7d8a330b-c407-496e-ba6b-1a3c14f9366f","Type":"ContainerDied","Data":"1ee0608ee9916a443c1aa4ebaa7aab850a6c1345c33edb8d7ceaf2f08e9aa734"} Nov 02 04:02:20.090282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.090233 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee0608ee9916a443c1aa4ebaa7aab850a6c1345c33edb8d7ceaf2f08e9aa734" Nov 02 04:02:20.418772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.418743 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp"] Nov 02 04:02:20.421919 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:20.421893 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-61c297-openshift-default-6b8dc99857-xmdbp"] Nov 02 04:02:22.014739 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.014704 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx"] Nov 02 04:02:22.015255 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.015239 4868 cpu_manager.go:414] "containerMap: removing container" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" Nov 02 04:02:22.015333 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.015256 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" Nov 02 04:02:22.015333 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.015289 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7d8a330b-c407-496e-ba6b-1a3c14f9366f" containerName="revoke-keys" Nov 02 04:02:22.015333 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.015295 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8a330b-c407-496e-ba6b-1a3c14f9366f" containerName="revoke-keys" Nov 02 04:02:22.015533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.015388 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" Nov 02 04:02:22.015533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.015406 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8a330b-c407-496e-ba6b-1a3c14f9366f" containerName="revoke-keys" Nov 02 04:02:22.018268 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.018247 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.114782 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.114753 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" path="/var/lib/kubelet/pods/80ae4893-94e4-484c-bac3-c1f582c7d863/volumes" Nov 02 04:02:22.120465 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.120442 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:02:22.120677 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.120661 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:02:22.123536 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.123515 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx"] Nov 02 04:02:22.145858 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.145833 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7wd\" (UniqueName: \"kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.145947 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.145884 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.145993 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.145981 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.247321 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.247298 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7wd\" (UniqueName: \"kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.247426 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.247346 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.247478 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.247429 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.248069 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.248046 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.249819 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.249793 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.304227 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.304177 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7wd\" (UniqueName: \"kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.331502 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.331483 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:22.708418 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:22.708394 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx"] Nov 02 04:02:22.710636 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:02:22.710605 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bb72c8_4733_4b05_a748_86f7af5d1cda.slice/crio-a283d462026e1406209e491eb2dc8d5c51c2365901914aed08bcf411252f9568 WatchSource:0}: Error finding container a283d462026e1406209e491eb2dc8d5c51c2365901914aed08bcf411252f9568: Status 404 returned error can't find the container with id a283d462026e1406209e491eb2dc8d5c51c2365901914aed08bcf411252f9568 Nov 02 04:02:23.110952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:23.110918 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" event={"ID":"23bb72c8-4733-4b05-a748-86f7af5d1cda","Type":"ContainerStarted","Data":"1cbe7153c24a8a8751d6bfb044af630456c55d2fb6e84091e79c980d05aefd93"} Nov 02 04:02:23.111289 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:23.110985 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" event={"ID":"23bb72c8-4733-4b05-a748-86f7af5d1cda","Type":"ContainerStarted","Data":"a283d462026e1406209e491eb2dc8d5c51c2365901914aed08bcf411252f9568"} Nov 02 04:02:23.223428 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:23.223382 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" podStartSLOduration=2.223367194 podStartE2EDuration="2.223367194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:02:23.222313095 +0000 UTC m=+4485.218014037" watchObservedRunningTime="2026-11-02 04:02:23.223367194 +0000 UTC m=+4485.219068129" Nov 02 04:02:24.118946 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:24.118918 4868 generic.go:356] "Generic (PLEG): container finished" podID="23bb72c8-4733-4b05-a748-86f7af5d1cda" containerID="1cbe7153c24a8a8751d6bfb044af630456c55d2fb6e84091e79c980d05aefd93" exitCode=0 Nov 02 04:02:24.119341 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:24.119006 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" event={"ID":"23bb72c8-4733-4b05-a748-86f7af5d1cda","Type":"ContainerDied","Data":"1cbe7153c24a8a8751d6bfb044af630456c55d2fb6e84091e79c980d05aefd93"} Nov 02 04:02:25.273067 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.273045 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:25.273600 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.273579 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd\" (UniqueName: \"kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd\") pod \"23bb72c8-4733-4b05-a748-86f7af5d1cda\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " Nov 02 04:02:25.273698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.273626 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca\") pod \"23bb72c8-4733-4b05-a748-86f7af5d1cda\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " Nov 02 04:02:25.273784 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.273735 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca\") pod \"23bb72c8-4733-4b05-a748-86f7af5d1cda\" (UID: \"23bb72c8-4733-4b05-a748-86f7af5d1cda\") " Nov 02 04:02:25.274098 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.274077 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca" pod "23bb72c8-4733-4b05-a748-86f7af5d1cda" (UID: "23bb72c8-4733-4b05-a748-86f7af5d1cda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:25.275554 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.275531 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd" pod "23bb72c8-4733-4b05-a748-86f7af5d1cda" (UID: "23bb72c8-4733-4b05-a748-86f7af5d1cda"). InnerVolumeSpecName "kube-api-access-mx7wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:25.275815 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.275799 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca" pod "23bb72c8-4733-4b05-a748-86f7af5d1cda" (UID: "23bb72c8-4733-4b05-a748-86f7af5d1cda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:02:25.392429 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.384271 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7wd\" (UniqueName: \"kubernetes.io/projected/23bb72c8-4733-4b05-a748-86f7af5d1cda-kube-api-access-mx7wd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:25.392429 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.384317 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:25.392429 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:25.384334 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23bb72c8-4733-4b05-a748-86f7af5d1cda-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:26.133250 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.133221 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" event={"ID":"23bb72c8-4733-4b05-a748-86f7af5d1cda","Type":"ContainerDied","Data":"a283d462026e1406209e491eb2dc8d5c51c2365901914aed08bcf411252f9568"} Nov 02 04:02:26.133250 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.133241 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx" Nov 02 04:02:26.133453 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.133263 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a283d462026e1406209e491eb2dc8d5c51c2365901914aed08bcf411252f9568" Nov 02 04:02:26.319069 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319036 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr"] Nov 02 04:02:26.319589 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319573 4868 cpu_manager.go:414] "containerMap: removing container" podUID="23bb72c8-4733-4b05-a748-86f7af5d1cda" containerName="revoke-keys" Nov 02 04:02:26.319640 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319590 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23bb72c8-4733-4b05-a748-86f7af5d1cda" containerName="revoke-keys" Nov 02 04:02:26.319640 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319608 4868 cpu_manager.go:414] "containerMap: removing container" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" Nov 02 04:02:26.319640 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319613 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" Nov 02 04:02:26.319740 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319697 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="23bb72c8-4733-4b05-a748-86f7af5d1cda" containerName="revoke-keys" Nov 02 04:02:26.319740 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.319710 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ae4893-94e4-484c-bac3-c1f582c7d863" containerName="istio-proxy" Nov 02 04:02:26.322106 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.322085 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.374949 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.374889 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:02:26.374949 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.374898 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:02:26.379230 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.379203 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr"] Nov 02 04:02:26.393181 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.393116 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744wt\" (UniqueName: \"kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.393322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.393303 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.393391 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.393374 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.494405 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.494374 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.494534 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.494425 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.494534 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.494515 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744wt\" (UniqueName: \"kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.495109 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.495083 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.496752 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.496729 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.550591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.550567 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-744wt\" (UniqueName: \"kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt\") pod \"maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:26.634114 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:26.634093 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:27.008365 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:27.008321 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr"] Nov 02 04:02:27.009335 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:02:27.009305 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4629ccab_e67b_4370_b7a6_ae3c941e56ac.slice/crio-c950c9fe6637820cd5e0626d2796cb09180c01dce975f274addb5f646351bac3 WatchSource:0}: Error finding container c950c9fe6637820cd5e0626d2796cb09180c01dce975f274addb5f646351bac3: Status 404 returned error can't find the container with id c950c9fe6637820cd5e0626d2796cb09180c01dce975f274addb5f646351bac3 Nov 02 04:02:27.180093 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:27.180052 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" event={"ID":"4629ccab-e67b-4370-b7a6-ae3c941e56ac","Type":"ContainerStarted","Data":"c950c9fe6637820cd5e0626d2796cb09180c01dce975f274addb5f646351bac3"} Nov 02 04:02:27.358497 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:27.358462 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6"] Nov 02 04:02:27.463262 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:27.463239 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls"] Nov 02 04:02:28.188754 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:28.188725 4868 generic.go:356] "Generic (PLEG): container finished" podID="4629ccab-e67b-4370-b7a6-ae3c941e56ac" containerID="225ad3964235473221db6a4bc9f1f8281553762c353ae3c39d130378204951c8" exitCode=0 Nov 02 04:02:28.188956 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:28.188790 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" event={"ID":"4629ccab-e67b-4370-b7a6-ae3c941e56ac","Type":"ContainerDied","Data":"225ad3964235473221db6a4bc9f1f8281553762c353ae3c39d130378204951c8"} Nov 02 04:02:29.348488 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.348463 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:29.432789 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.432761 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca\") pod \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " Nov 02 04:02:29.432934 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.432818 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt\" (UniqueName: \"kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt\") pod \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " Nov 02 04:02:29.432934 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.432843 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca\") pod \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\" (UID: \"4629ccab-e67b-4370-b7a6-ae3c941e56ac\") " Nov 02 04:02:29.433104 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.433078 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca" pod "4629ccab-e67b-4370-b7a6-ae3c941e56ac" (UID: "4629ccab-e67b-4370-b7a6-ae3c941e56ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:29.434886 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.434860 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca" pod "4629ccab-e67b-4370-b7a6-ae3c941e56ac" (UID: "4629ccab-e67b-4370-b7a6-ae3c941e56ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:02:29.435001 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.434982 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt" pod "4629ccab-e67b-4370-b7a6-ae3c941e56ac" (UID: "4629ccab-e67b-4370-b7a6-ae3c941e56ac"). InnerVolumeSpecName "kube-api-access-744wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:29.534014 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.533939 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-744wt\" (UniqueName: \"kubernetes.io/projected/4629ccab-e67b-4370-b7a6-ae3c941e56ac-kube-api-access-744wt\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:29.534014 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.533968 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:29.534014 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:29.533982 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4629ccab-e67b-4370-b7a6-ae3c941e56ac-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:30.202516 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:30.202488 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" Nov 02 04:02:30.202674 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:30.202517 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr" event={"ID":"4629ccab-e67b-4370-b7a6-ae3c941e56ac","Type":"ContainerDied","Data":"c950c9fe6637820cd5e0626d2796cb09180c01dce975f274addb5f646351bac3"} Nov 02 04:02:30.202674 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:30.202565 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c950c9fe6637820cd5e0626d2796cb09180c01dce975f274addb5f646351bac3" Nov 02 04:02:32.359733 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:32.359680 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" podUID="78799308-9296-4456-b7fc-4f00a6353a18" containerName="payload-processing" containerID="cri-o://229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab" gracePeriod=25 Nov 02 04:02:32.463670 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:32.463632 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" podUID="e367ea5b-449e-4185-9f60-436900d4981f" containerName="payload-pre-processing" containerID="cri-o://a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc" gracePeriod=25 Nov 02 04:02:33.447463 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:33.447412 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:02:38.162987 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:38.162941 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" podUID="e367ea5b-449e-4185-9f60-436900d4981f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:02:38.169546 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:38.169511 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" podUID="78799308-9296-4456-b7fc-4f00a6353a18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:02:38.209975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:38.209945 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw"] Nov 02 04:02:38.345698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:38.345662 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2"] Nov 02 04:02:38.699167 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:38.699140 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 04:02:38.710194 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:38.710169 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 04:02:39.866634 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:39.866604 4868 scope.go:122] "RemoveContainer" containerID="751b2fc7d36161168eb0eb07797f6ae98be84cf947f340dbca4723f4586c2eaa" Nov 02 04:02:39.878775 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:39.878703 4868 scope.go:122] "RemoveContainer" containerID="5b19d25ae30fe96e539c8318d413cc7d7a3f6dcd8dcaf452186ad4e00ac597a1" Nov 02 04:02:39.912360 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:39.912326 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx"] Nov 02 04:02:39.915199 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:39.915177 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee0fed-pqvrx"] Nov 02 04:02:40.115866 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:40.115832 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23bb72c8-4733-4b05-a748-86f7af5d1cda" path="/var/lib/kubelet/pods/23bb72c8-4733-4b05-a748-86f7af5d1cda/volumes" Nov 02 04:02:43.210754 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.210690 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" podUID="ccf645cb-7832-4037-964e-512b37c92926" containerName="payload-processing" containerID="cri-o://0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d" gracePeriod=25 Nov 02 04:02:43.346736 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.346705 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" podUID="8274f7b0-d23f-4200-870f-252676b95f5f" containerName="payload-pre-processing" containerID="cri-o://f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267" gracePeriod=25 Nov 02 04:02:43.474410 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.474388 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 04:02:43.592059 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.592034 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 04:02:43.592487 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.592456 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8\" (UniqueName: \"kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8\") pod \"ccf645cb-7832-4037-964e-512b37c92926\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " Nov 02 04:02:43.592642 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.592580 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume\") pod \"ccf645cb-7832-4037-964e-512b37c92926\" (UID: \"ccf645cb-7832-4037-964e-512b37c92926\") " Nov 02 04:02:43.593012 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.592987 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume" pod "ccf645cb-7832-4037-964e-512b37c92926" (UID: "ccf645cb-7832-4037-964e-512b37c92926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:43.594604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.594577 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8" pod "ccf645cb-7832-4037-964e-512b37c92926" (UID: "ccf645cb-7832-4037-964e-512b37c92926"). InnerVolumeSpecName "kube-api-access-7fgx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:43.694260 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.694228 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume\") pod \"8274f7b0-d23f-4200-870f-252676b95f5f\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " Nov 02 04:02:43.694430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.694318 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8\" (UniqueName: \"kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8\") pod \"8274f7b0-d23f-4200-870f-252676b95f5f\" (UID: \"8274f7b0-d23f-4200-870f-252676b95f5f\") " Nov 02 04:02:43.694602 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.694579 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume" pod "8274f7b0-d23f-4200-870f-252676b95f5f" (UID: "8274f7b0-d23f-4200-870f-252676b95f5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:43.694781 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.694762 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fgx8\" (UniqueName: \"kubernetes.io/projected/ccf645cb-7832-4037-964e-512b37c92926-kube-api-access-7fgx8\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:43.694842 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.694788 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8274f7b0-d23f-4200-870f-252676b95f5f-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:43.694842 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.694803 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf645cb-7832-4037-964e-512b37c92926-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:43.696214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.696196 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8" pod "8274f7b0-d23f-4200-870f-252676b95f5f" (UID: "8274f7b0-d23f-4200-870f-252676b95f5f"). InnerVolumeSpecName "kube-api-access-jbgz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:43.796232 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:43.796166 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbgz8\" (UniqueName: \"kubernetes.io/projected/8274f7b0-d23f-4200-870f-252676b95f5f-kube-api-access-jbgz8\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:44.317925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.317890 4868 generic.go:356] "Generic (PLEG): container finished" podID="ccf645cb-7832-4037-964e-512b37c92926" containerID="0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d" exitCode=0 Nov 02 04:02:44.318422 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.317935 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" Nov 02 04:02:44.318422 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.317975 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" event={"ID":"ccf645cb-7832-4037-964e-512b37c92926","Type":"ContainerDied","Data":"0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d"} Nov 02 04:02:44.318422 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.318026 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw" event={"ID":"ccf645cb-7832-4037-964e-512b37c92926","Type":"ContainerDied","Data":"cbad238ba9d7a4d1f7aef7c3cea1af28a90c79beaabe02a410322337a3c60b47"} Nov 02 04:02:44.318422 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.318053 4868 scope.go:122] "RemoveContainer" containerID="0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d" Nov 02 04:02:44.319569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.319538 4868 generic.go:356] "Generic (PLEG): container finished" podID="8274f7b0-d23f-4200-870f-252676b95f5f" containerID="f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267" exitCode=0 Nov 02 04:02:44.319657 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.319606 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" event={"ID":"8274f7b0-d23f-4200-870f-252676b95f5f","Type":"ContainerDied","Data":"f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267"} Nov 02 04:02:44.319657 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.319621 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" Nov 02 04:02:44.319657 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.319639 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2" event={"ID":"8274f7b0-d23f-4200-870f-252676b95f5f","Type":"ContainerDied","Data":"db051ce0fe9984c3b6a58fc06f8999f5019882c9c76bb94b6df40eed9c3be3b1"} Nov 02 04:02:44.327895 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.327879 4868 scope.go:122] "RemoveContainer" containerID="0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d" Nov 02 04:02:44.328173 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:44.328154 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d\": container with ID starting with 0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d not found: ID does not exist" containerID="0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d" Nov 02 04:02:44.328255 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.328183 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d"} err="failed to get container status \"0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d\": rpc error: code = NotFound desc = could not find container \"0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d\": container with ID starting with 0cdb58f8f4297f4cfb0c87ae28a80e5afcddc2e9ba1f2d50e4b209fc8b6b3e3d not found: ID does not exist" Nov 02 04:02:44.328255 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.328208 4868 scope.go:122] "RemoveContainer" containerID="f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267" Nov 02 04:02:44.337296 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.337278 4868 scope.go:122] "RemoveContainer" containerID="f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267" Nov 02 04:02:44.337562 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:44.337546 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267\": container with ID starting with f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267 not found: ID does not exist" containerID="f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267" Nov 02 04:02:44.337617 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.337570 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267"} err="failed to get container status \"f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267\": rpc error: code = NotFound desc = could not find container \"f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267\": container with ID starting with f23b61aa69133d4d83b0819e7df743cdfdb660c480d08df394f68a88efa3f267 not found: ID does not exist" Nov 02 04:02:44.481765 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.481683 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2"] Nov 02 04:02:44.484474 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.484453 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e69cf2-6b598d9bf9-62jj2"] Nov 02 04:02:44.594816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.594792 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw"] Nov 02 04:02:44.599499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:44.599476 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e69cf2-795748cc99-f2hgw"] Nov 02 04:02:46.114680 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:46.114648 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8274f7b0-d23f-4200-870f-252676b95f5f" path="/var/lib/kubelet/pods/8274f7b0-d23f-4200-870f-252676b95f5f/volumes" Nov 02 04:02:46.115167 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:46.115150 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf645cb-7832-4037-964e-512b37c92926" path="/var/lib/kubelet/pods/ccf645cb-7832-4037-964e-512b37c92926/volumes" Nov 02 04:02:46.614191 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:46.614158 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l"] Nov 02 04:02:46.614431 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:46.614393 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" containerID="cri-o://c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5" gracePeriod=30 Nov 02 04:02:47.402283 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402249 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8"] Nov 02 04:02:47.402767 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402753 4868 cpu_manager.go:414] "containerMap: removing container" podUID="8274f7b0-d23f-4200-870f-252676b95f5f" containerName="payload-pre-processing" Nov 02 04:02:47.402816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402768 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8274f7b0-d23f-4200-870f-252676b95f5f" containerName="payload-pre-processing" Nov 02 04:02:47.402816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402779 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4629ccab-e67b-4370-b7a6-ae3c941e56ac" containerName="revoke-keys" Nov 02 04:02:47.402816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402785 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4629ccab-e67b-4370-b7a6-ae3c941e56ac" containerName="revoke-keys" Nov 02 04:02:47.402816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402805 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ccf645cb-7832-4037-964e-512b37c92926" containerName="payload-processing" Nov 02 04:02:47.402816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402810 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf645cb-7832-4037-964e-512b37c92926" containerName="payload-processing" Nov 02 04:02:47.402975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402908 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="8274f7b0-d23f-4200-870f-252676b95f5f" containerName="payload-pre-processing" Nov 02 04:02:47.402975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402915 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4629ccab-e67b-4370-b7a6-ae3c941e56ac" containerName="revoke-keys" Nov 02 04:02:47.402975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.402924 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf645cb-7832-4037-964e-512b37c92926" containerName="payload-processing" Nov 02 04:02:47.406942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.406915 4868 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 04:02:47.407031 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.406973 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 04:02:47.407580 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.407564 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.461100 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.461074 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8"] Nov 02 04:02:47.507474 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.507447 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:02:47.507474 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.507464 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:02:47.547799 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.547775 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.547909 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.547839 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpsv\" (UniqueName: \"kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.547981 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.547962 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.648626 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.648602 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpsv\" (UniqueName: \"kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.648714 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.648665 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.648774 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.648717 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.649313 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.649292 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.651058 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.651036 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.705378 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.705302 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpsv\" (UniqueName: \"kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:47.719923 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:47.719904 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:48.089734 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:48.089696 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8"] Nov 02 04:02:48.090338 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:02:48.090303 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aafe44a_15f1_4821_8254_61c596e742f3.slice/crio-55d01add51895ba91575f15e74c8fa930586d703394fe664df3ad88ca6233575 WatchSource:0}: Error finding container 55d01add51895ba91575f15e74c8fa930586d703394fe664df3ad88ca6233575: Status 404 returned error can't find the container with id 55d01add51895ba91575f15e74c8fa930586d703394fe664df3ad88ca6233575 Nov 02 04:02:48.136951 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:48.136922 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" podUID="78799308-9296-4456-b7fc-4f00a6353a18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:02:48.137053 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:48.136926 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" podUID="e367ea5b-449e-4185-9f60-436900d4981f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:02:48.347253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:48.347217 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" event={"ID":"4aafe44a-15f1-4821-8254-61c596e742f3","Type":"ContainerStarted","Data":"55d01add51895ba91575f15e74c8fa930586d703394fe664df3ad88ca6233575"} Nov 02 04:02:49.356291 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:49.356253 4868 generic.go:356] "Generic (PLEG): container finished" podID="4aafe44a-15f1-4821-8254-61c596e742f3" containerID="260e50f66388af44c551e916f6041d15b840a3cd40549232ae97eae5a1c9c036" exitCode=0 Nov 02 04:02:49.356803 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:49.356309 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" event={"ID":"4aafe44a-15f1-4821-8254-61c596e742f3","Type":"ContainerDied","Data":"260e50f66388af44c551e916f6041d15b840a3cd40549232ae97eae5a1c9c036"} Nov 02 04:02:50.504291 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.504262 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:50.688756 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.688685 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca\") pod \"4aafe44a-15f1-4821-8254-61c596e742f3\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " Nov 02 04:02:50.688908 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.688863 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca\") pod \"4aafe44a-15f1-4821-8254-61c596e742f3\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " Nov 02 04:02:50.688986 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.688946 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv\" (UniqueName: \"kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv\") pod \"4aafe44a-15f1-4821-8254-61c596e742f3\" (UID: \"4aafe44a-15f1-4821-8254-61c596e742f3\") " Nov 02 04:02:50.689225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.689194 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca" pod "4aafe44a-15f1-4821-8254-61c596e742f3" (UID: "4aafe44a-15f1-4821-8254-61c596e742f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:50.689392 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.689307 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:50.690788 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.690766 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca" pod "4aafe44a-15f1-4821-8254-61c596e742f3" (UID: "4aafe44a-15f1-4821-8254-61c596e742f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:02:50.690921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.690900 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv" pod "4aafe44a-15f1-4821-8254-61c596e742f3" (UID: "4aafe44a-15f1-4821-8254-61c596e742f3"). InnerVolumeSpecName "kube-api-access-vxpsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:50.789912 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.789886 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpsv\" (UniqueName: \"kubernetes.io/projected/4aafe44a-15f1-4821-8254-61c596e742f3-kube-api-access-vxpsv\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:50.789912 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:50.789911 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aafe44a-15f1-4821-8254-61c596e742f3-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:51.230932 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:51.230899 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc"] Nov 02 04:02:51.234864 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:51.234835 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae1315-mf4nc"] Nov 02 04:02:51.369650 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:51.369625 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" Nov 02 04:02:51.369810 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:51.369623 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8" event={"ID":"4aafe44a-15f1-4821-8254-61c596e742f3","Type":"ContainerDied","Data":"55d01add51895ba91575f15e74c8fa930586d703394fe664df3ad88ca6233575"} Nov 02 04:02:51.369810 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:51.369719 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d01add51895ba91575f15e74c8fa930586d703394fe664df3ad88ca6233575" Nov 02 04:02:51.863231 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:51.863208 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 04:02:52.004759 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004685 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.004904 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004755 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.004904 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004846 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.004904 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004877 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.005103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004908 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.005103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004959 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.005103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.004979 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:52.005103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005044 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:52.005103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005052 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.005103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005093 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.005476 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005155 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy\") pod \"968ffdd6-532a-4921-a7cc-b9a3a940580d\" (UID: \"968ffdd6-532a-4921-a7cc-b9a3a940580d\") " Nov 02 04:02:52.005476 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005237 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:52.005589 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005548 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:52.005765 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005746 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.005861 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005769 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.005861 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005771 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:52.005861 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005783 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ffdd6-532a-4921-a7cc-b9a3a940580d-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.005861 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.005816 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.007723 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.007697 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:02:52.007830 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.007719 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:52.007905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.007851 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "kube-api-access-k889w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:52.008020 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.007997 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy" pod "968ffdd6-532a-4921-a7cc-b9a3a940580d" (UID: "968ffdd6-532a-4921-a7cc-b9a3a940580d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:02:52.106447 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.106422 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.106447 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.106447 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k889w\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-kube-api-access-k889w\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.106624 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.106457 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.106624 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.106466 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.106624 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.106474 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ffdd6-532a-4921-a7cc-b9a3a940580d-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.115856 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.115829 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21029201-19f2-4c38-85c6-997809d0fd71" path="/var/lib/kubelet/pods/21029201-19f2-4c38-85c6-997809d0fd71/volumes" Nov 02 04:02:52.378185 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.378157 4868 generic.go:356] "Generic (PLEG): container finished" podID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerID="c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5" exitCode=0 Nov 02 04:02:52.378303 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.378207 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" Nov 02 04:02:52.378303 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.378239 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" event={"ID":"968ffdd6-532a-4921-a7cc-b9a3a940580d","Type":"ContainerDied","Data":"c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5"} Nov 02 04:02:52.378303 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.378298 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l" event={"ID":"968ffdd6-532a-4921-a7cc-b9a3a940580d","Type":"ContainerDied","Data":"41fca3369a304868fc8b2451ed829014b1f4de2108026b12fb5edabd95bde43c"} Nov 02 04:02:52.378500 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.378332 4868 scope.go:122] "RemoveContainer" containerID="c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5" Nov 02 04:02:52.472563 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.472499 4868 scope.go:122] "RemoveContainer" containerID="2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398" Nov 02 04:02:52.543487 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.543457 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l"] Nov 02 04:02:52.577961 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.577934 4868 scope.go:122] "RemoveContainer" containerID="c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5" Nov 02 04:02:52.578427 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:52.578395 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5\": container with ID starting with c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5 not found: ID does not exist" containerID="c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5" Nov 02 04:02:52.578551 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.578437 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5"} err="failed to get container status \"c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5\": rpc error: code = NotFound desc = could not find container \"c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5\": container with ID starting with c1f8bbd73801621894009a570753888e9a93ea73e42d9313bf74be5e5d7a27e5 not found: ID does not exist" Nov 02 04:02:52.578551 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.578461 4868 scope.go:122] "RemoveContainer" containerID="2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398" Nov 02 04:02:52.578861 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:52.578834 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398\": container with ID starting with 2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398 not found: ID does not exist" containerID="2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398" Nov 02 04:02:52.578961 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.578870 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398"} err="failed to get container status \"2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398\": rpc error: code = NotFound desc = could not find container \"2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398\": container with ID starting with 2c5323764773e7a2275006dee8dcfbb561d19b5583978070c4710048e346d398 not found: ID does not exist" Nov 02 04:02:52.594840 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.593443 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee0fed-openshift-default-689c8ff78f-znd9l"] Nov 02 04:02:52.605036 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.605016 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 04:02:52.610751 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.610731 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume\") pod \"78799308-9296-4456-b7fc-4f00a6353a18\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " Nov 02 04:02:52.610842 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.610790 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69\" (UniqueName: \"kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69\") pod \"78799308-9296-4456-b7fc-4f00a6353a18\" (UID: \"78799308-9296-4456-b7fc-4f00a6353a18\") " Nov 02 04:02:52.611176 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.611148 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume" pod "78799308-9296-4456-b7fc-4f00a6353a18" (UID: "78799308-9296-4456-b7fc-4f00a6353a18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:52.611276 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.611265 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78799308-9296-4456-b7fc-4f00a6353a18-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.612793 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.612773 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69" pod "78799308-9296-4456-b7fc-4f00a6353a18" (UID: "78799308-9296-4456-b7fc-4f00a6353a18"). InnerVolumeSpecName "kube-api-access-lcj69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:52.712138 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.712110 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcj69\" (UniqueName: \"kubernetes.io/projected/78799308-9296-4456-b7fc-4f00a6353a18-kube-api-access-lcj69\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.759797 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.759774 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 04:02:52.813418 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.813391 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume\") pod \"e367ea5b-449e-4185-9f60-436900d4981f\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " Nov 02 04:02:52.813551 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.813435 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd\" (UniqueName: \"kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd\") pod \"e367ea5b-449e-4185-9f60-436900d4981f\" (UID: \"e367ea5b-449e-4185-9f60-436900d4981f\") " Nov 02 04:02:52.813741 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.813712 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume" pod "e367ea5b-449e-4185-9f60-436900d4981f" (UID: "e367ea5b-449e-4185-9f60-436900d4981f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:02:52.813872 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.813855 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e367ea5b-449e-4185-9f60-436900d4981f-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:52.815501 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.815478 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd" pod "e367ea5b-449e-4185-9f60-436900d4981f" (UID: "e367ea5b-449e-4185-9f60-436900d4981f"). InnerVolumeSpecName "kube-api-access-sf9qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:02:52.914696 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:52.914638 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9qd\" (UniqueName: \"kubernetes.io/projected/e367ea5b-449e-4185-9f60-436900d4981f-kube-api-access-sf9qd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:02:53.386834 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.386801 4868 generic.go:356] "Generic (PLEG): container finished" podID="78799308-9296-4456-b7fc-4f00a6353a18" containerID="229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab" exitCode=0 Nov 02 04:02:53.386999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.386855 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" Nov 02 04:02:53.386999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.386886 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" event={"ID":"78799308-9296-4456-b7fc-4f00a6353a18","Type":"ContainerDied","Data":"229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab"} Nov 02 04:02:53.386999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.386947 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6" event={"ID":"78799308-9296-4456-b7fc-4f00a6353a18","Type":"ContainerDied","Data":"f1d037d833a0fd3956b0c52d65549b964063652b44e70e52c1295481d23bac1e"} Nov 02 04:02:53.386999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.386983 4868 scope.go:122] "RemoveContainer" containerID="229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab" Nov 02 04:02:53.390066 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.390036 4868 generic.go:356] "Generic (PLEG): container finished" podID="e367ea5b-449e-4185-9f60-436900d4981f" containerID="a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc" exitCode=0 Nov 02 04:02:53.390185 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.390106 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" Nov 02 04:02:53.390185 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.390131 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" event={"ID":"e367ea5b-449e-4185-9f60-436900d4981f","Type":"ContainerDied","Data":"a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc"} Nov 02 04:02:53.390282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.390198 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls" event={"ID":"e367ea5b-449e-4185-9f60-436900d4981f","Type":"ContainerDied","Data":"a6efc49e73a418f5b94335e5a54e85f931ad8789f72c70869d82e882664ab793"} Nov 02 04:02:53.397747 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.397717 4868 scope.go:122] "RemoveContainer" containerID="229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab" Nov 02 04:02:53.398075 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:53.398055 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab\": container with ID starting with 229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab not found: ID does not exist" containerID="229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab" Nov 02 04:02:53.398184 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.398086 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab"} err="failed to get container status \"229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab\": rpc error: code = NotFound desc = could not find container \"229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab\": container with ID starting with 229e368699cd23277e257210f47ff4700468b466b3a6aaf9bec8620da4d26eab not found: ID does not exist" Nov 02 04:02:53.398184 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.398110 4868 scope.go:122] "RemoveContainer" containerID="a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc" Nov 02 04:02:53.408072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.408054 4868 scope.go:122] "RemoveContainer" containerID="a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc" Nov 02 04:02:53.408386 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:02:53.408365 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc\": container with ID starting with a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc not found: ID does not exist" containerID="a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc" Nov 02 04:02:53.408450 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.408391 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc"} err="failed to get container status \"a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc\": rpc error: code = NotFound desc = could not find container \"a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc\": container with ID starting with a57c134d4866b0ca1da5124fa68ae56126715a919dad186f81282282dea095cc not found: ID does not exist" Nov 02 04:02:53.550315 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.550285 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6"] Nov 02 04:02:53.556480 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.556453 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c3ea5-845f45b9c5-s7qg6"] Nov 02 04:02:53.664032 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.663965 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls"] Nov 02 04:02:53.667997 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:53.667973 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c3ea5-577dd95747-jqdls"] Nov 02 04:02:54.114323 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:54.114289 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78799308-9296-4456-b7fc-4f00a6353a18" path="/var/lib/kubelet/pods/78799308-9296-4456-b7fc-4f00a6353a18/volumes" Nov 02 04:02:54.114852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:54.114797 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" path="/var/lib/kubelet/pods/968ffdd6-532a-4921-a7cc-b9a3a940580d/volumes" Nov 02 04:02:54.115178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:54.115153 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e367ea5b-449e-4185-9f60-436900d4981f" path="/var/lib/kubelet/pods/e367ea5b-449e-4185-9f60-436900d4981f/volumes" Nov 02 04:02:55.045084 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:55.045051 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v"] Nov 02 04:02:55.094482 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:55.094454 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2bfaea-mnl5v"] Nov 02 04:02:56.114614 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:56.114579 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70eea3d2-c0b0-44d4-a7b2-7078d918f9e8" path="/var/lib/kubelet/pods/70eea3d2-c0b0-44d4-a7b2-7078d918f9e8/volumes" Nov 02 04:02:57.043953 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:57.043919 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5"] Nov 02 04:02:57.093113 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:57.093085 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a53e5fb-tcwl5"] Nov 02 04:02:57.574163 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:57.574131 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn"] Nov 02 04:02:57.574753 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:57.574373 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" containerID="cri-o://f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753" gracePeriod=30 Nov 02 04:02:58.116209 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:58.116179 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9045f2f7-7e64-4a29-b7fa-d4c369063e5b" path="/var/lib/kubelet/pods/9045f2f7-7e64-4a29-b7fa-d4c369063e5b/volumes" Nov 02 04:02:59.730469 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730434 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg"] Nov 02 04:02:59.730952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730904 4868 cpu_manager.go:414] "containerMap: removing container" podUID="78799308-9296-4456-b7fc-4f00a6353a18" containerName="payload-processing" Nov 02 04:02:59.730952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730917 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78799308-9296-4456-b7fc-4f00a6353a18" containerName="payload-processing" Nov 02 04:02:59.730952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730937 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e367ea5b-449e-4185-9f60-436900d4981f" containerName="payload-pre-processing" Nov 02 04:02:59.730952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730942 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e367ea5b-449e-4185-9f60-436900d4981f" containerName="payload-pre-processing" Nov 02 04:02:59.730952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730953 4868 cpu_manager.go:414] "containerMap: removing container" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730958 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730981 4868 cpu_manager.go:414] "containerMap: removing container" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.730986 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.731000 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4aafe44a-15f1-4821-8254-61c596e742f3" containerName="revoke-keys" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.731005 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aafe44a-15f1-4821-8254-61c596e742f3" containerName="revoke-keys" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.731085 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.731095 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aafe44a-15f1-4821-8254-61c596e742f3" containerName="revoke-keys" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.731103 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="78799308-9296-4456-b7fc-4f00a6353a18" containerName="payload-processing" Nov 02 04:02:59.731130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.731111 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e367ea5b-449e-4185-9f60-436900d4981f" containerName="payload-pre-processing" Nov 02 04:02:59.736142 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.736117 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.786931 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.786904 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.787050 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.786956 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.787050 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.787022 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq49c\" (UniqueName: \"kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.789203 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.789029 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:02:59.791347 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.791326 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg"] Nov 02 04:02:59.837607 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.837585 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:02:59.887673 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.887646 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.887790 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.887689 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq49c\" (UniqueName: \"kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.887844 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.887804 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.888370 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.888332 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.890196 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.890166 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:02:59.943776 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:02:59.943750 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq49c\" (UniqueName: \"kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:03:00.048893 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:00.048834 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:03:00.422468 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:00.422442 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg"] Nov 02 04:03:00.424073 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:03:00.424043 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d82439_f3de_4f61_8d48_e8acff8b2ccd.slice/crio-ce6106c3aabc9ced35bdb6c128bf921296ff3f542942f3fe91dc6c6196c7d058 WatchSource:0}: Error finding container ce6106c3aabc9ced35bdb6c128bf921296ff3f542942f3fe91dc6c6196c7d058: Status 404 returned error can't find the container with id ce6106c3aabc9ced35bdb6c128bf921296ff3f542942f3fe91dc6c6196c7d058 Nov 02 04:03:00.451290 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:00.451257 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" event={"ID":"b5d82439-f3de-4f61-8d48-e8acff8b2ccd","Type":"ContainerStarted","Data":"ce6106c3aabc9ced35bdb6c128bf921296ff3f542942f3fe91dc6c6196c7d058"} Nov 02 04:03:01.459516 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:01.459483 4868 generic.go:356] "Generic (PLEG): container finished" podID="b5d82439-f3de-4f61-8d48-e8acff8b2ccd" containerID="bb9016d71fcd37cd22626b9ac4c3ae53bab1544082dc172659a8a0c41137d2be" exitCode=0 Nov 02 04:03:01.459924 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:01.459571 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" event={"ID":"b5d82439-f3de-4f61-8d48-e8acff8b2ccd","Type":"ContainerDied","Data":"bb9016d71fcd37cd22626b9ac4c3ae53bab1544082dc172659a8a0c41137d2be"} Nov 02 04:03:02.669890 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.669866 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:03:02.718414 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.718345 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca\") pod \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " Nov 02 04:03:02.718743 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.718457 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c\" (UniqueName: \"kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c\") pod \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " Nov 02 04:03:02.718743 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.718497 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca\") pod \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\" (UID: \"b5d82439-f3de-4f61-8d48-e8acff8b2ccd\") " Nov 02 04:03:02.718933 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.718909 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca" pod "b5d82439-f3de-4f61-8d48-e8acff8b2ccd" (UID: "b5d82439-f3de-4f61-8d48-e8acff8b2ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:03:02.720780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.720758 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca" pod "b5d82439-f3de-4f61-8d48-e8acff8b2ccd" (UID: "b5d82439-f3de-4f61-8d48-e8acff8b2ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:03:02.720860 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.720783 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c" pod "b5d82439-f3de-4f61-8d48-e8acff8b2ccd" (UID: "b5d82439-f3de-4f61-8d48-e8acff8b2ccd"). InnerVolumeSpecName "kube-api-access-gq49c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:02.817133 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.817109 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 04:03:02.819834 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.819815 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.819894 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.819838 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq49c\" (UniqueName: \"kubernetes.io/projected/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-kube-api-access-gq49c\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.819894 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.819849 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d82439-f3de-4f61-8d48-e8acff8b2ccd-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.921039 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921015 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921082 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921104 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921124 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921163 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921212 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921253 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921476 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921430 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921476 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921450 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:02.921601 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921480 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data\") pod \"24607332-f862-4be8-9712-8e1e7f6c1b68\" (UID: \"24607332-f862-4be8-9712-8e1e7f6c1b68\") " Nov 02 04:03:02.921926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921675 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:02.921926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921796 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:02.921926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921903 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:02.922101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.921928 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:03:02.922101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.922011 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.922101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.922033 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.922101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.922047 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.922101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.922062 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:02.923235 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.923217 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:02.923323 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.923305 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:03:02.923668 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.923649 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "kube-api-access-tshzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:02.923729 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:02.923662 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy" pod "24607332-f862-4be8-9712-8e1e7f6c1b68" (UID: "24607332-f862-4be8-9712-8e1e7f6c1b68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:03.023388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.023307 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:03.023388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.023334 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tshzc\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-kube-api-access-tshzc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:03.023388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.023344 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:03.023388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.023376 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24607332-f862-4be8-9712-8e1e7f6c1b68-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:03.023388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.023385 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24607332-f862-4be8-9712-8e1e7f6c1b68-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:03.446474 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:03:03.446439 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:03:03.471971 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.471947 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" Nov 02 04:03:03.472125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.471973 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg" event={"ID":"b5d82439-f3de-4f61-8d48-e8acff8b2ccd","Type":"ContainerDied","Data":"ce6106c3aabc9ced35bdb6c128bf921296ff3f542942f3fe91dc6c6196c7d058"} Nov 02 04:03:03.472125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.472023 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce6106c3aabc9ced35bdb6c128bf921296ff3f542942f3fe91dc6c6196c7d058" Nov 02 04:03:03.473792 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.473768 4868 generic.go:356] "Generic (PLEG): container finished" podID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerID="f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753" exitCode=0 Nov 02 04:03:03.473921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.473821 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" Nov 02 04:03:03.473921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.473832 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" event={"ID":"24607332-f862-4be8-9712-8e1e7f6c1b68","Type":"ContainerDied","Data":"f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753"} Nov 02 04:03:03.473921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.473874 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn" event={"ID":"24607332-f862-4be8-9712-8e1e7f6c1b68","Type":"ContainerDied","Data":"b86262d5af105e7db56edd1d592afdf3695179cd78457a137d50de83a18c83f2"} Nov 02 04:03:03.473921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.473896 4868 scope.go:122] "RemoveContainer" containerID="f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753" Nov 02 04:03:03.484948 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.484929 4868 scope.go:122] "RemoveContainer" containerID="3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068" Nov 02 04:03:03.494426 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.494406 4868 scope.go:122] "RemoveContainer" containerID="f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753" Nov 02 04:03:03.494660 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:03:03.494643 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753\": container with ID starting with f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753 not found: ID does not exist" containerID="f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753" Nov 02 04:03:03.494711 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.494677 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753"} err="failed to get container status \"f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753\": rpc error: code = NotFound desc = could not find container \"f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753\": container with ID starting with f66b92bd2cdc930ac838eb0014de873bd901163acbcb0fa6053fe3018fd72753 not found: ID does not exist" Nov 02 04:03:03.494711 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.494692 4868 scope.go:122] "RemoveContainer" containerID="3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068" Nov 02 04:03:03.494910 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:03:03.494894 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068\": container with ID starting with 3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068 not found: ID does not exist" containerID="3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068" Nov 02 04:03:03.494953 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.494914 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068"} err="failed to get container status \"3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068\": rpc error: code = NotFound desc = could not find container \"3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068\": container with ID starting with 3edb520034cecbc0f3ece18ebc9683d82c59e2a334a9cd6a53eed25353a0b068 not found: ID does not exist" Nov 02 04:03:03.641687 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.641661 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn"] Nov 02 04:03:03.691181 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:03.691157 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae1315-openshift-default-568d79d5b5-pd7pn"] Nov 02 04:03:04.115863 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:04.115831 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" path="/var/lib/kubelet/pods/24607332-f862-4be8-9712-8e1e7f6c1b68/volumes" Nov 02 04:03:05.045643 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:05.045608 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs"] Nov 02 04:03:05.089917 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:05.089888 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073cbf6d-vn6bs"] Nov 02 04:03:06.121867 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:06.121834 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e694303-3925-4b9a-9fec-5e6aa7baaa91" path="/var/lib/kubelet/pods/0e694303-3925-4b9a-9fec-5e6aa7baaa91/volumes" Nov 02 04:03:08.044883 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:08.044837 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt"] Nov 02 04:03:08.094599 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:08.094566 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ffe1e6-sjkwt"] Nov 02 04:03:08.114714 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:08.114680 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beaf4ac9-f2bc-414d-a7d5-c7af035ca68a" path="/var/lib/kubelet/pods/beaf4ac9-f2bc-414d-a7d5-c7af035ca68a/volumes" Nov 02 04:03:26.478844 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:26.478812 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg"] Nov 02 04:03:26.527891 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:26.527864 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-61f3b6-44lwg"] Nov 02 04:03:28.114805 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:28.114771 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2756e5-3d00-4a27-bf63-e71f13714b64" path="/var/lib/kubelet/pods/2e2756e5-3d00-4a27-bf63-e71f13714b64/volumes" Nov 02 04:03:33.510067 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:03:33.510029 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:03:35.501464 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501426 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2"] Nov 02 04:03:35.501901 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501884 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b5d82439-f3de-4f61-8d48-e8acff8b2ccd" containerName="revoke-keys" Nov 02 04:03:35.501992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501902 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d82439-f3de-4f61-8d48-e8acff8b2ccd" containerName="revoke-keys" Nov 02 04:03:35.501992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501942 4868 cpu_manager.go:414] "containerMap: removing container" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" Nov 02 04:03:35.501992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501948 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" Nov 02 04:03:35.501992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501961 4868 cpu_manager.go:414] "containerMap: removing container" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" Nov 02 04:03:35.501992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.501967 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" Nov 02 04:03:35.502152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.502035 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="968ffdd6-532a-4921-a7cc-b9a3a940580d" containerName="istio-proxy" Nov 02 04:03:35.502152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.502046 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" Nov 02 04:03:35.502152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.502053 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="24607332-f862-4be8-9712-8e1e7f6c1b68" containerName="istio-proxy" Nov 02 04:03:35.502152 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.502060 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d82439-f3de-4f61-8d48-e8acff8b2ccd" containerName="revoke-keys" Nov 02 04:03:35.506562 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.506539 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.562170 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.562142 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2"] Nov 02 04:03:35.608205 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.608180 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:03:35.608388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.608291 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:03:35.651458 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.651428 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.651605 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.651477 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgsj\" (UniqueName: \"kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.651605 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.651585 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.754597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.754503 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.771372 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.767494 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgsj\" (UniqueName: \"kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.771372 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.767616 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.771372 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.770849 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.779891 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.779860 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:35.825830 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:35.825802 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgsj\" (UniqueName: \"kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:36.121438 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:36.121406 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:36.492796 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:36.492770 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2"] Nov 02 04:03:36.495115 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:03:36.495086 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbecf9b_3aef_41eb_94c1_fd6c1f4e72f2.slice/crio-0f4ba9049ed352e75dc8507aa9ac30f589a30cd01eb3f2e8bad9327991e7e00c WatchSource:0}: Error finding container 0f4ba9049ed352e75dc8507aa9ac30f589a30cd01eb3f2e8bad9327991e7e00c: Status 404 returned error can't find the container with id 0f4ba9049ed352e75dc8507aa9ac30f589a30cd01eb3f2e8bad9327991e7e00c Nov 02 04:03:36.680533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:36.680491 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" event={"ID":"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2","Type":"ContainerStarted","Data":"0f4ba9049ed352e75dc8507aa9ac30f589a30cd01eb3f2e8bad9327991e7e00c"} Nov 02 04:03:37.501008 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:37.500974 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5"] Nov 02 04:03:37.550366 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:37.550326 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c3ea5-mznf5"] Nov 02 04:03:37.697541 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:37.697462 4868 generic.go:356] "Generic (PLEG): container finished" podID="3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" containerID="ee145596beb0dfdc9ad6861fd563b96037c4490c71c762cc9d4c4d9eb7884e3c" exitCode=0 Nov 02 04:03:37.697916 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:37.697545 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" event={"ID":"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2","Type":"ContainerDied","Data":"ee145596beb0dfdc9ad6861fd563b96037c4490c71c762cc9d4c4d9eb7884e3c"} Nov 02 04:03:38.115735 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.115705 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8a330b-c407-496e-ba6b-1a3c14f9366f" path="/var/lib/kubelet/pods/7d8a330b-c407-496e-ba6b-1a3c14f9366f/volumes" Nov 02 04:03:38.849762 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.849735 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:38.903877 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.903843 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca\") pod \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " Nov 02 04:03:38.903999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.903982 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj\" (UniqueName: \"kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj\") pod \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " Nov 02 04:03:38.904097 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.904081 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca\") pod \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\" (UID: \"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2\") " Nov 02 04:03:38.904497 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.904469 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca" pod "3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" (UID: "3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:03:38.906024 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.906000 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj" pod "3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" (UID: "3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2"). InnerVolumeSpecName "kube-api-access-qpgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:38.906140 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:38.906023 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca" pod "3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" (UID: "3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:03:39.006011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.005939 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:39.006011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.005968 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:39.006011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.005978 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgsj\" (UniqueName: \"kubernetes.io/projected/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2-kube-api-access-qpgsj\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:39.710961 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.710929 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" Nov 02 04:03:39.711146 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.710972 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2" event={"ID":"3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2","Type":"ContainerDied","Data":"0f4ba9049ed352e75dc8507aa9ac30f589a30cd01eb3f2e8bad9327991e7e00c"} Nov 02 04:03:39.711146 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.711025 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4ba9049ed352e75dc8507aa9ac30f589a30cd01eb3f2e8bad9327991e7e00c" Nov 02 04:03:39.916805 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.916778 4868 scope.go:122] "RemoveContainer" containerID="4d872f2e54c42b77a52d79ab1e8cded314ba5376c7b894e3d51a9b741176aea3" Nov 02 04:03:39.926129 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:39.926110 4868 scope.go:122] "RemoveContainer" containerID="709dd89a0df07941901f067bce8465aea670e6431ca035ceeaa07aa9a2154dd9" Nov 02 04:03:43.785428 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:43.785395 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh"] Nov 02 04:03:43.785921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:43.785644 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" containerID="cri-o://35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca" gracePeriod=30 Nov 02 04:03:44.600576 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.600545 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl"] Nov 02 04:03:44.600979 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.600965 4868 cpu_manager.go:414] "containerMap: removing container" podUID="3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" containerName="revoke-keys" Nov 02 04:03:44.601037 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.600981 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" containerName="revoke-keys" Nov 02 04:03:44.601102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.601092 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" containerName="revoke-keys" Nov 02 04:03:44.604215 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.604197 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.658210 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.658186 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:03:44.661189 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.661163 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl"] Nov 02 04:03:44.666584 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.666561 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.666700 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.666599 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rwf\" (UniqueName: \"kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.666770 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.666702 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.707562 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.707541 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:03:44.767731 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.767708 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.767836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.767745 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rwf\" (UniqueName: \"kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.767836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.767781 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.768447 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.768420 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.770173 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.770155 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.824014 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.823990 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rwf\" (UniqueName: \"kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:44.917440 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:44.917419 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:45.285998 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:45.285971 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl"] Nov 02 04:03:45.287418 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:03:45.287391 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b3afc5_3263_49ab_8502_6f43f8cc8d0c.slice/crio-2f5590e70595e325bd8b8ea5fbe375161264bb0dd9b6c4c531b80f74bb3cf2d0 WatchSource:0}: Error finding container 2f5590e70595e325bd8b8ea5fbe375161264bb0dd9b6c4c531b80f74bb3cf2d0: Status 404 returned error can't find the container with id 2f5590e70595e325bd8b8ea5fbe375161264bb0dd9b6c4c531b80f74bb3cf2d0 Nov 02 04:03:45.751788 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:45.751754 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" event={"ID":"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c","Type":"ContainerStarted","Data":"64bf6eee434cbaa7ab1f4d09fbb1f275ed84e793e856a08b2143c50c4b04ab26"} Nov 02 04:03:45.751958 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:45.751819 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" event={"ID":"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c","Type":"ContainerStarted","Data":"2f5590e70595e325bd8b8ea5fbe375161264bb0dd9b6c4c531b80f74bb3cf2d0"} Nov 02 04:03:45.865800 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:45.865751 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" podStartSLOduration=1.8657359260000002 podStartE2EDuration="1.865735926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:03:45.864305192 +0000 UTC m=+4567.860006140" watchObservedRunningTime="2026-11-02 04:03:45.865735926 +0000 UTC m=+4567.861436860" Nov 02 04:03:46.759841 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:46.759811 4868 generic.go:356] "Generic (PLEG): container finished" podID="c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" containerID="64bf6eee434cbaa7ab1f4d09fbb1f275ed84e793e856a08b2143c50c4b04ab26" exitCode=0 Nov 02 04:03:46.760023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:46.759894 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" event={"ID":"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c","Type":"ContainerDied","Data":"64bf6eee434cbaa7ab1f4d09fbb1f275ed84e793e856a08b2143c50c4b04ab26"} Nov 02 04:03:47.898451 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:47.898422 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:47.998883 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:47.998860 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf\" (UniqueName: \"kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf\") pod \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " Nov 02 04:03:47.999062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:47.999041 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca\") pod \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " Nov 02 04:03:47.999235 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:47.999218 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca\") pod \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\" (UID: \"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c\") " Nov 02 04:03:47.999427 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:47.999407 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca" pod "c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" (UID: "c0b3afc5-3263-49ab-8502-6f43f8cc8d0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:03:47.999694 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:47.999677 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:48.001010 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.000991 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf" pod "c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" (UID: "c0b3afc5-3263-49ab-8502-6f43f8cc8d0c"). InnerVolumeSpecName "kube-api-access-56rwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:48.001127 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.001107 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca" pod "c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" (UID: "c0b3afc5-3263-49ab-8502-6f43f8cc8d0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:03:48.100753 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.100725 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:48.100874 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.100774 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rwf\" (UniqueName: \"kubernetes.io/projected/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c-kube-api-access-56rwf\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:48.776802 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.776768 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" Nov 02 04:03:48.776964 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.776772 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl" event={"ID":"c0b3afc5-3263-49ab-8502-6f43f8cc8d0c","Type":"ContainerDied","Data":"2f5590e70595e325bd8b8ea5fbe375161264bb0dd9b6c4c531b80f74bb3cf2d0"} Nov 02 04:03:48.776964 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:48.776908 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5590e70595e325bd8b8ea5fbe375161264bb0dd9b6c4c531b80f74bb3cf2d0" Nov 02 04:03:49.045257 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.045229 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 04:03:49.110312 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110286 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110456 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110331 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110500 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110598 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110557 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110598 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110585 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110704 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110669 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110767 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110746 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:49.110822 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110790 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110948 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110827 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110948 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110862 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token\") pod \"d8da2353-de4d-449c-9000-de0e4632bf69\" (UID: \"d8da2353-de4d-449c-9000-de0e4632bf69\") " Nov 02 04:03:49.110948 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.110916 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:49.111119 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111065 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:49.111119 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111098 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:49.111304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111284 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.111496 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111474 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.111582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111499 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.111582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111513 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.111582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.111439 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:03:49.112864 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.112839 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "kube-api-access-q7s68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:49.112975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.112926 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:03:49.113030 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.113007 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:03:49.113079 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.113011 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token" pod "d8da2353-de4d-449c-9000-de0e4632bf69" (UID: "d8da2353-de4d-449c-9000-de0e4632bf69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:03:49.212925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.212899 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da2353-de4d-449c-9000-de0e4632bf69-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.213054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.212926 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.213054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.212992 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7s68\" (UniqueName: \"kubernetes.io/projected/d8da2353-de4d-449c-9000-de0e4632bf69-kube-api-access-q7s68\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.213054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.213011 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da2353-de4d-449c-9000-de0e4632bf69-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.213054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.213025 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da2353-de4d-449c-9000-de0e4632bf69-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:03:49.785716 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.785677 4868 generic.go:356] "Generic (PLEG): container finished" podID="d8da2353-de4d-449c-9000-de0e4632bf69" containerID="35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca" exitCode=0 Nov 02 04:03:49.785895 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.785740 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" Nov 02 04:03:49.785895 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.785760 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" event={"ID":"d8da2353-de4d-449c-9000-de0e4632bf69","Type":"ContainerDied","Data":"35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca"} Nov 02 04:03:49.785895 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.785818 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh" event={"ID":"d8da2353-de4d-449c-9000-de0e4632bf69","Type":"ContainerDied","Data":"c60b25db557f7d5b5e3748d94cbb6a71955177c1a9da2498a3d6ee4dd54cd5c6"} Nov 02 04:03:49.785895 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.785840 4868 scope.go:122] "RemoveContainer" containerID="35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca" Nov 02 04:03:49.796104 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.796082 4868 scope.go:122] "RemoveContainer" containerID="7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0" Nov 02 04:03:49.804697 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.804679 4868 scope.go:122] "RemoveContainer" containerID="35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca" Nov 02 04:03:49.804984 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:03:49.804967 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca\": container with ID starting with 35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca not found: ID does not exist" containerID="35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca" Nov 02 04:03:49.805042 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.804993 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca"} err="failed to get container status \"35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca\": rpc error: code = NotFound desc = could not find container \"35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca\": container with ID starting with 35dbac4170b0e294eec4470510eff7ecf6f02d0289c7223c06f96463fdd8ceca not found: ID does not exist" Nov 02 04:03:49.805042 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.805013 4868 scope.go:122] "RemoveContainer" containerID="7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0" Nov 02 04:03:49.805223 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:03:49.805208 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0\": container with ID starting with 7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0 not found: ID does not exist" containerID="7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0" Nov 02 04:03:49.805265 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:49.805226 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0"} err="failed to get container status \"7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0\": rpc error: code = NotFound desc = could not find container \"7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0\": container with ID starting with 7179c7235f7b3f35f084af1a0f4bd01e86a3f8702ba8ba00e9cf67da884faed0 not found: ID does not exist" Nov 02 04:03:50.000561 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:50.000531 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh"] Nov 02 04:03:50.051335 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:50.051271 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4c3ea5-openshift-default-5b8ddbdc45-sqjsh"] Nov 02 04:03:50.115460 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:50.115428 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" path="/var/lib/kubelet/pods/d8da2353-de4d-449c-9000-de0e4632bf69/volumes" Nov 02 04:03:54.356860 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:54.356820 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr"] Nov 02 04:03:54.405801 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:54.405774 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e69cf2-rbsrr"] Nov 02 04:03:55.551176 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:55.551145 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg"] Nov 02 04:03:55.600759 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:55.600731 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-a7cf93-f8hwg"] Nov 02 04:03:56.114715 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:56.114684 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4629ccab-e67b-4370-b7a6-ae3c941e56ac" path="/var/lib/kubelet/pods/4629ccab-e67b-4370-b7a6-ae3c941e56ac/volumes" Nov 02 04:03:56.115128 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:56.115111 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d82439-f3de-4f61-8d48-e8acff8b2ccd" path="/var/lib/kubelet/pods/b5d82439-f3de-4f61-8d48-e8acff8b2ccd/volumes" Nov 02 04:03:56.896363 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:56.896318 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8"] Nov 02 04:03:56.947467 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:56.947432 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-147104-lp4j8"] Nov 02 04:03:56.977763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:56.977732 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x"] Nov 02 04:03:57.966544 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:57.966512 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2"] Nov 02 04:03:57.968638 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:57.968616 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7c1bee-5wzs2"] Nov 02 04:03:58.114293 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:58.114262 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2" path="/var/lib/kubelet/pods/3fbecf9b-3aef-41eb-94c1-fd6c1f4e72f2/volumes" Nov 02 04:03:58.114708 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:03:58.114683 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aafe44a-15f1-4821-8254-61c596e742f3" path="/var/lib/kubelet/pods/4aafe44a-15f1-4821-8254-61c596e742f3/volumes" Nov 02 04:04:00.718975 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:00.718945 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7"] Nov 02 04:04:00.719416 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:00.719149 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" containerID="cri-o://2b2c5be07ffd457513f818fd424447e1c686266bb9c52c52b12aa9a03b2c20f7" gracePeriod=30 Nov 02 04:04:01.978180 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:01.978125 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerName="payload-pre-processing" containerID="cri-o://1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705" gracePeriod=25 Nov 02 04:04:02.898730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.898686 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8"] Nov 02 04:04:02.899112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899097 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" containerName="revoke-keys" Nov 02 04:04:02.899112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899111 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" containerName="revoke-keys" Nov 02 04:04:02.899253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899127 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" Nov 02 04:04:02.899253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899132 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" Nov 02 04:04:02.899253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899222 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" Nov 02 04:04:02.899253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899232 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" containerName="revoke-keys" Nov 02 04:04:02.899253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.899239 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" Nov 02 04:04:02.904139 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.904107 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:02.959346 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.959304 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9"] Nov 02 04:04:02.960319 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.960289 4868 cpu_manager.go:414] "containerMap: removing container" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" Nov 02 04:04:02.960319 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.960317 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8da2353-de4d-449c-9000-de0e4632bf69" containerName="istio-proxy" Nov 02 04:04:02.967679 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:02.967643 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.009537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.009491 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8"] Nov 02 04:04:03.010054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.009741 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:04:03.060501 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.060464 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3f24d0-openshift-default-dockercfg-8smsm\"" Nov 02 04:04:03.063132 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.061405 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:04:03.068425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.068397 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9"] Nov 02 04:04:03.076397 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076365 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcct\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.076516 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076491 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.076563 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076544 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj44l\" (UniqueName: \"kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.076621 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076596 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.076787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076768 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.076876 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076804 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.076876 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076852 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.076993 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076945 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.076993 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.076975 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.077124 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.077095 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.077178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.077135 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.077259 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.077239 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.178733 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178635 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.178733 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178684 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj44l\" (UniqueName: \"kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.178733 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178732 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178775 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178803 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178850 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178886 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178916 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.178990 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.179023 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179456 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.179078 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179456 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.179093 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179456 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.179110 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcct\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179751 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.179724 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.179879 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.179853 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.180101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.180074 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.180183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.180117 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.180251 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.180217 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.181579 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.181553 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.181709 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.181589 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.184860 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.184828 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.250381 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.249047 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcct\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.250381 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.249091 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj44l\" (UniqueName: \"kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.252807 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.252777 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token\") pod \"e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.283631 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.283604 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:03.448466 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:03.448381 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:04:03.517183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.517155 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:03.712084 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.712060 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9"] Nov 02 04:04:03.714367 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:03.714325 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2693bf6_5c99_45ed_bed7_605704705d8f.slice/crio-2cfc7e5391fabe41a8e7f29218ff116096c71e2f141a6d23ad805f36207c984f WatchSource:0}: Error finding container 2cfc7e5391fabe41a8e7f29218ff116096c71e2f141a6d23ad805f36207c984f: Status 404 returned error can't find the container with id 2cfc7e5391fabe41a8e7f29218ff116096c71e2f141a6d23ad805f36207c984f Nov 02 04:04:03.716893 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.716860 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:04:03.716982 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.716928 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:04:03.716982 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.716954 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:04:03.883732 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.883697 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" event={"ID":"e2693bf6-5c99-45ed-bed7-605704705d8f","Type":"ContainerStarted","Data":"6dbc89d926e8146143175e592a9128962fa902c7c499db1dbd72857b0843bf63"} Nov 02 04:04:03.883914 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.883750 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" event={"ID":"e2693bf6-5c99-45ed-bed7-605704705d8f","Type":"ContainerStarted","Data":"2cfc7e5391fabe41a8e7f29218ff116096c71e2f141a6d23ad805f36207c984f"} Nov 02 04:04:03.927576 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:03.927547 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8"] Nov 02 04:04:03.928449 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:03.928424 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbbf7b4b_4e08_4d39_8e1f_982045f55275.slice/crio-5f3ae954ea2c9c15eee34456b3cc3cbb1c1446ce1c6739c84e707026a0660e32 WatchSource:0}: Error finding container 5f3ae954ea2c9c15eee34456b3cc3cbb1c1446ce1c6739c84e707026a0660e32: Status 404 returned error can't find the container with id 5f3ae954ea2c9c15eee34456b3cc3cbb1c1446ce1c6739c84e707026a0660e32 Nov 02 04:04:04.002635 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.002516 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podStartSLOduration=2.002497711 podStartE2EDuration="2.002497711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:04.00183312 +0000 UTC m=+4585.997534064" watchObservedRunningTime="2026-11-02 04:04:04.002497711 +0000 UTC m=+4585.998198655" Nov 02 04:04:04.284010 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.283943 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:04.289056 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.289034 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:04.746719 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.746687 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2"] Nov 02 04:04:04.750226 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.750210 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:04.808438 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.808363 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2"] Nov 02 04:04:04.890614 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.890587 4868 generic.go:356] "Generic (PLEG): container finished" podID="dbbf7b4b-4e08-4d39-8e1f-982045f55275" containerID="f8dfd2a78c514d884eefac4234f0b155bb833e7b762e41ff5185b2d215f00dcf" exitCode=0 Nov 02 04:04:04.890763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.890670 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" event={"ID":"dbbf7b4b-4e08-4d39-8e1f-982045f55275","Type":"ContainerDied","Data":"f8dfd2a78c514d884eefac4234f0b155bb833e7b762e41ff5185b2d215f00dcf"} Nov 02 04:04:04.890763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.890726 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" event={"ID":"dbbf7b4b-4e08-4d39-8e1f-982045f55275","Type":"ContainerStarted","Data":"5f3ae954ea2c9c15eee34456b3cc3cbb1c1446ce1c6739c84e707026a0660e32"} Nov 02 04:04:04.891102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.891085 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:04.892235 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.892215 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:04:04.903342 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.903319 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:04.903485 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.903465 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59k6\" (UniqueName: \"kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:04.903589 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:04.903572 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.018435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.007706 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.018435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.007803 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59k6\" (UniqueName: \"kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.018435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.007932 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.018435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.009638 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.018435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.012148 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.058991 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.058958 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr"] Nov 02 04:04:05.059190 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.059171 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" containerID="cri-o://d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a" gracePeriod=30 Nov 02 04:04:05.063993 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.063968 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59k6\" (UniqueName: \"kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6\") pod \"maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.363618 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.363587 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:05.735387 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.735345 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2"] Nov 02 04:04:05.737916 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:05.737889 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088baf26_62c0_4fec_968f_084f1a2723e5.slice/crio-fc2c91608620bf1c2e3b95cffafc160a587d6c4eee08eef35c8d163aeb26d252 WatchSource:0}: Error finding container fc2c91608620bf1c2e3b95cffafc160a587d6c4eee08eef35c8d163aeb26d252: Status 404 returned error can't find the container with id fc2c91608620bf1c2e3b95cffafc160a587d6c4eee08eef35c8d163aeb26d252 Nov 02 04:04:05.901610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.901581 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz"] Nov 02 04:04:05.904617 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.904593 4868 generic.go:356] "Generic (PLEG): container finished" podID="01774e87-b148-4801-b4a8-7bf0564cd846" containerID="2b2c5be07ffd457513f818fd424447e1c686266bb9c52c52b12aa9a03b2c20f7" exitCode=0 Nov 02 04:04:05.905874 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.905845 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" event={"ID":"088baf26-62c0-4fec-968f-084f1a2723e5","Type":"ContainerStarted","Data":"fc2c91608620bf1c2e3b95cffafc160a587d6c4eee08eef35c8d163aeb26d252"} Nov 02 04:04:05.905986 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.905895 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" event={"ID":"01774e87-b148-4801-b4a8-7bf0564cd846","Type":"ContainerDied","Data":"2b2c5be07ffd457513f818fd424447e1c686266bb9c52c52b12aa9a03b2c20f7"} Nov 02 04:04:05.905986 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.905936 4868 scope.go:122] "RemoveContainer" containerID="bb60c3eb662b9c8789a7f70d10296c68079ca715b87193c358919c464cf6fc29" Nov 02 04:04:05.906105 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.906055 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:05.918380 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.918325 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:05.918839 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.918654 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:05.918933 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.918884 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2th9w\" (UniqueName: \"kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:05.960921 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.960700 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz"] Nov 02 04:04:05.975684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:05.975659 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 04:04:06.019558 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019470 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.019704 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019643 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.019771 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019685 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.019771 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019747 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.019915 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019807 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.019915 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019861 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.019915 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.019903 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020027 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020069 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020101 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert\") pod \"01774e87-b148-4801-b4a8-7bf0564cd846\" (UID: \"01774e87-b148-4801-b4a8-7bf0564cd846\") " Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020321 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2th9w\" (UniqueName: \"kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020402 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020480 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.020588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020555 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.020988 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.020776 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:06.021507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.021091 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.021507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.021126 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:06.021507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.021394 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:06.021824 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.021791 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:06.024033 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.024008 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.024124 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.024055 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "kube-api-access-fsl6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:06.024124 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.024114 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:04:06.026259 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.026228 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:06.026388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.026327 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy" pod "01774e87-b148-4801-b4a8-7bf0564cd846" (UID: "01774e87-b148-4801-b4a8-7bf0564cd846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:06.051966 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.051950 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:06.077185 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.077163 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2th9w\" (UniqueName: \"kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.121456 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121431 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca\") pod \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " Nov 02 04:04:06.121568 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121509 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca\") pod \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " Nov 02 04:04:06.121568 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121546 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l\" (UniqueName: \"kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l\") pod \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\" (UID: \"dbbf7b4b-4e08-4d39-8e1f-982045f55275\") " Nov 02 04:04:06.121762 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121746 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsl6f\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-kube-api-access-fsl6f\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121764 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121775 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01774e87-b148-4801-b4a8-7bf0564cd846-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121788 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121796 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121808 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01774e87-b148-4801-b4a8-7bf0564cd846-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121818 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01774e87-b148-4801-b4a8-7bf0564cd846-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121831 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01774e87-b148-4801-b4a8-7bf0564cd846-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.121852 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.121834 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca" pod "dbbf7b4b-4e08-4d39-8e1f-982045f55275" (UID: "dbbf7b4b-4e08-4d39-8e1f-982045f55275"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:06.123585 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.123563 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l" pod "dbbf7b4b-4e08-4d39-8e1f-982045f55275" (UID: "dbbf7b4b-4e08-4d39-8e1f-982045f55275"). InnerVolumeSpecName "kube-api-access-lj44l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:06.123671 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.123601 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca" pod "dbbf7b4b-4e08-4d39-8e1f-982045f55275" (UID: "dbbf7b4b-4e08-4d39-8e1f-982045f55275"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:06.223137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.223108 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.223137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.223139 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbf7b4b-4e08-4d39-8e1f-982045f55275-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.223297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.223150 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj44l\" (UniqueName: \"kubernetes.io/projected/dbbf7b4b-4e08-4d39-8e1f-982045f55275-kube-api-access-lj44l\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:06.230131 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.230109 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:06.606478 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.606455 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz"] Nov 02 04:04:06.608405 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:06.608381 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125eb568_0bc5_431b_a994_dcadb7eaa3de.slice/crio-e33d4b74de16917d8f0c422a31c00bdcaadc9fc153ab937ad73b12c1dc5fcc72 WatchSource:0}: Error finding container e33d4b74de16917d8f0c422a31c00bdcaadc9fc153ab937ad73b12c1dc5fcc72: Status 404 returned error can't find the container with id e33d4b74de16917d8f0c422a31c00bdcaadc9fc153ab937ad73b12c1dc5fcc72 Nov 02 04:04:06.912454 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.912378 4868 generic.go:356] "Generic (PLEG): container finished" podID="088baf26-62c0-4fec-968f-084f1a2723e5" containerID="fe4cebfe762285b897b9637340444b71bd20fa8b9a36bc1a57f0a462141738a0" exitCode=0 Nov 02 04:04:06.912621 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.912473 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" event={"ID":"088baf26-62c0-4fec-968f-084f1a2723e5","Type":"ContainerDied","Data":"fe4cebfe762285b897b9637340444b71bd20fa8b9a36bc1a57f0a462141738a0"} Nov 02 04:04:06.913929 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.913905 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" Nov 02 04:04:06.914056 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.913913 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8" event={"ID":"dbbf7b4b-4e08-4d39-8e1f-982045f55275","Type":"ContainerDied","Data":"5f3ae954ea2c9c15eee34456b3cc3cbb1c1446ce1c6739c84e707026a0660e32"} Nov 02 04:04:06.914056 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.914029 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f3ae954ea2c9c15eee34456b3cc3cbb1c1446ce1c6739c84e707026a0660e32" Nov 02 04:04:06.915412 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.915391 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" Nov 02 04:04:06.915551 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.915410 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7" event={"ID":"01774e87-b148-4801-b4a8-7bf0564cd846","Type":"ContainerDied","Data":"a4972e9798cedf0fd81598e56c9add53671b0d8c127f41d813cac3af1153c5c5"} Nov 02 04:04:06.915551 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.915456 4868 scope.go:122] "RemoveContainer" containerID="2b2c5be07ffd457513f818fd424447e1c686266bb9c52c52b12aa9a03b2c20f7" Nov 02 04:04:06.917769 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:06.917648 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" event={"ID":"125eb568-0bc5-431b-a994-dcadb7eaa3de","Type":"ContainerStarted","Data":"e33d4b74de16917d8f0c422a31c00bdcaadc9fc153ab937ad73b12c1dc5fcc72"} Nov 02 04:04:07.199494 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:07.199421 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7"] Nov 02 04:04:07.202524 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:07.202500 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-e69cf2-openshift-default-78ffd9b88f-n2bd7"] Nov 02 04:04:07.926302 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:07.926269 4868 generic.go:356] "Generic (PLEG): container finished" podID="125eb568-0bc5-431b-a994-dcadb7eaa3de" containerID="6614cc357da8790d0a4bf18f210441309fd45022acc2bdd7c44be3f2645c10a5" exitCode=0 Nov 02 04:04:07.926783 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:07.926372 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" event={"ID":"125eb568-0bc5-431b-a994-dcadb7eaa3de","Type":"ContainerDied","Data":"6614cc357da8790d0a4bf18f210441309fd45022acc2bdd7c44be3f2645c10a5"} Nov 02 04:04:08.070317 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.070290 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:08.114050 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.114024 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" path="/var/lib/kubelet/pods/01774e87-b148-4801-b4a8-7bf0564cd846/volumes" Nov 02 04:04:08.141518 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.141495 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca\") pod \"088baf26-62c0-4fec-968f-084f1a2723e5\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " Nov 02 04:04:08.141672 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.141590 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca\") pod \"088baf26-62c0-4fec-968f-084f1a2723e5\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " Nov 02 04:04:08.141672 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.141637 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6\" (UniqueName: \"kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6\") pod \"088baf26-62c0-4fec-968f-084f1a2723e5\" (UID: \"088baf26-62c0-4fec-968f-084f1a2723e5\") " Nov 02 04:04:08.141974 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.141952 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca" pod "088baf26-62c0-4fec-968f-084f1a2723e5" (UID: "088baf26-62c0-4fec-968f-084f1a2723e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:08.143348 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.143329 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca" pod "088baf26-62c0-4fec-968f-084f1a2723e5" (UID: "088baf26-62c0-4fec-968f-084f1a2723e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:08.143525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.143502 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6" pod "088baf26-62c0-4fec-968f-084f1a2723e5" (UID: "088baf26-62c0-4fec-968f-084f1a2723e5"). InnerVolumeSpecName "kube-api-access-v59k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:08.207524 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.207453 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:04:08.243062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.243033 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:08.243062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.243063 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59k6\" (UniqueName: \"kubernetes.io/projected/088baf26-62c0-4fec-968f-084f1a2723e5-kube-api-access-v59k6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:08.243238 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.243077 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088baf26-62c0-4fec-968f-084f1a2723e5-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:08.933671 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.933640 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" Nov 02 04:04:08.934050 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.933637 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2" event={"ID":"088baf26-62c0-4fec-968f-084f1a2723e5","Type":"ContainerDied","Data":"fc2c91608620bf1c2e3b95cffafc160a587d6c4eee08eef35c8d163aeb26d252"} Nov 02 04:04:08.934050 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:08.933769 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2c91608620bf1c2e3b95cffafc160a587d6c4eee08eef35c8d163aeb26d252" Nov 02 04:04:09.077517 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.077489 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:09.152236 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.152209 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca\") pod \"125eb568-0bc5-431b-a994-dcadb7eaa3de\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " Nov 02 04:04:09.152441 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.152243 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w\" (UniqueName: \"kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w\") pod \"125eb568-0bc5-431b-a994-dcadb7eaa3de\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " Nov 02 04:04:09.152441 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.152269 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca\") pod \"125eb568-0bc5-431b-a994-dcadb7eaa3de\" (UID: \"125eb568-0bc5-431b-a994-dcadb7eaa3de\") " Nov 02 04:04:09.152642 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.152618 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca" pod "125eb568-0bc5-431b-a994-dcadb7eaa3de" (UID: "125eb568-0bc5-431b-a994-dcadb7eaa3de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:09.152816 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.152787 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:09.154446 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.154415 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca" pod "125eb568-0bc5-431b-a994-dcadb7eaa3de" (UID: "125eb568-0bc5-431b-a994-dcadb7eaa3de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:09.154539 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.154512 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w" pod "125eb568-0bc5-431b-a994-dcadb7eaa3de" (UID: "125eb568-0bc5-431b-a994-dcadb7eaa3de"). InnerVolumeSpecName "kube-api-access-2th9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:09.243483 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.243415 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5"] Nov 02 04:04:09.247080 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.247045 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg"] Nov 02 04:04:09.254249 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.254222 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2th9w\" (UniqueName: \"kubernetes.io/projected/125eb568-0bc5-431b-a994-dcadb7eaa3de-kube-api-access-2th9w\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:09.254249 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.254250 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125eb568-0bc5-431b-a994-dcadb7eaa3de-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:09.941143 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.941106 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" event={"ID":"125eb568-0bc5-431b-a994-dcadb7eaa3de","Type":"ContainerDied","Data":"e33d4b74de16917d8f0c422a31c00bdcaadc9fc153ab937ad73b12c1dc5fcc72"} Nov 02 04:04:09.941548 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.941184 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33d4b74de16917d8f0c422a31c00bdcaadc9fc153ab937ad73b12c1dc5fcc72" Nov 02 04:04:09.941548 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:09.941122 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz" Nov 02 04:04:10.383850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.383826 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 04:04:10.473116 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473054 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473116 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473107 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473326 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473137 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473326 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473195 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473326 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473264 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473326 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473292 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473326 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473292 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:10.473649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473332 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473386 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473494 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token\") pod \"bb801bbf-9cd2-489e-985f-cd17ef2638de\" (UID: \"bb801bbf-9cd2-489e-985f-cd17ef2638de\") " Nov 02 04:04:10.473649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473612 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:10.473649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473627 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:10.473937 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473683 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:10.473937 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.473812 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:10.474044 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.474027 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.474102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.474046 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.474102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.474062 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.474102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.474077 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb801bbf-9cd2-489e-985f-cd17ef2638de-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.474102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.474092 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.475610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.475587 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "kube-api-access-fqwm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:10.475610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.475590 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:04:10.475900 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.475872 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:10.475969 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.475929 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token" pod "bb801bbf-9cd2-489e-985f-cd17ef2638de" (UID: "bb801bbf-9cd2-489e-985f-cd17ef2638de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:10.574706 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.574670 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.574706 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.574702 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.574706 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.574713 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqwm6\" (UniqueName: \"kubernetes.io/projected/bb801bbf-9cd2-489e-985f-cd17ef2638de-kube-api-access-fqwm6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.574951 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.574723 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb801bbf-9cd2-489e-985f-cd17ef2638de-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:10.608683 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.608652 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md"] Nov 02 04:04:10.609073 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609058 4868 cpu_manager.go:414] "containerMap: removing container" podUID="dbbf7b4b-4e08-4d39-8e1f-982045f55275" containerName="revoke-keys" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609073 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbf7b4b-4e08-4d39-8e1f-982045f55275" containerName="revoke-keys" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609094 4868 cpu_manager.go:414] "containerMap: removing container" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609099 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609114 4868 cpu_manager.go:414] "containerMap: removing container" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609119 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609130 4868 cpu_manager.go:414] "containerMap: removing container" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" Nov 02 04:04:10.609137 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609135 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609147 4868 cpu_manager.go:414] "containerMap: removing container" podUID="088baf26-62c0-4fec-968f-084f1a2723e5" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609152 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088baf26-62c0-4fec-968f-084f1a2723e5" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609160 4868 cpu_manager.go:414] "containerMap: removing container" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609164 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609174 4868 cpu_manager.go:414] "containerMap: removing container" podUID="125eb568-0bc5-431b-a994-dcadb7eaa3de" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609178 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125eb568-0bc5-431b-a994-dcadb7eaa3de" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609241 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="088baf26-62c0-4fec-968f-084f1a2723e5" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609250 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609257 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="125eb568-0bc5-431b-a994-dcadb7eaa3de" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609264 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbf7b4b-4e08-4d39-8e1f-982045f55275" containerName="revoke-keys" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609270 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" Nov 02 04:04:10.609373 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.609277 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerName="istio-proxy" Nov 02 04:04:10.612770 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.612744 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.665154 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.665126 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3f24d0\"" Nov 02 04:04:10.665602 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.665270 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3f24d0-dockercfg-ttfzd\"" Nov 02 04:04:10.668602 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.668576 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md"] Nov 02 04:04:10.675978 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.675953 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.676114 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.676053 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wd4f\" (UniqueName: \"kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f\") pod \"payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.777494 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.777426 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.777637 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.777533 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd4f\" (UniqueName: \"kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f\") pod \"payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.778218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.778193 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.834021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.834000 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd4f\" (UniqueName: \"kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f\") pod \"payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.927255 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.927232 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:10.949537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.949507 4868 generic.go:356] "Generic (PLEG): container finished" podID="bb801bbf-9cd2-489e-985f-cd17ef2638de" containerID="d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a" exitCode=0 Nov 02 04:04:10.949942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.949582 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" Nov 02 04:04:10.949942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.949628 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" event={"ID":"bb801bbf-9cd2-489e-985f-cd17ef2638de","Type":"ContainerDied","Data":"d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a"} Nov 02 04:04:10.949942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.949697 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr" event={"ID":"bb801bbf-9cd2-489e-985f-cd17ef2638de","Type":"ContainerDied","Data":"7ef0e464a5ccf2a4cc152b9689751918860f90390732e0d1aa9d201f93b97595"} Nov 02 04:04:10.949942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.949744 4868 scope.go:122] "RemoveContainer" containerID="d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a" Nov 02 04:04:10.961285 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.961268 4868 scope.go:122] "RemoveContainer" containerID="55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386" Nov 02 04:04:10.971563 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.971543 4868 scope.go:122] "RemoveContainer" containerID="d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a" Nov 02 04:04:10.971870 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:10.971839 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a\": container with ID starting with d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a not found: ID does not exist" containerID="d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a" Nov 02 04:04:10.971980 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.971882 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a"} err="failed to get container status \"d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a\": rpc error: code = NotFound desc = could not find container \"d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a\": container with ID starting with d2a02dd700deffa935a594674f271e61cafebd258c98ed90b69bf1c71251d98a not found: ID does not exist" Nov 02 04:04:10.971980 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.971919 4868 scope.go:122] "RemoveContainer" containerID="55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386" Nov 02 04:04:10.972219 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:10.972192 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386\": container with ID starting with 55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386 not found: ID does not exist" containerID="55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386" Nov 02 04:04:10.972276 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:10.972230 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386"} err="failed to get container status \"55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386\": rpc error: code = NotFound desc = could not find container \"55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386\": container with ID starting with 55631360a5cf5097def5c27f46b70aa9fc014d907b264729b697ac943180a386 not found: ID does not exist" Nov 02 04:04:11.114800 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:11.114769 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr"] Nov 02 04:04:11.120304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:11.120278 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7c1bee-openshift-default-8b9cd95c5-pg8tr"] Nov 02 04:04:11.301463 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:11.301438 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md"] Nov 02 04:04:11.303650 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:11.303620 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be50840_13ca_4a52_a9c1_f159af4e9312.slice/crio-7c9ae346dacbec23f20df450be35156d7dc35aad369e7b1284cc48101df60155 WatchSource:0}: Error finding container 7c9ae346dacbec23f20df450be35156d7dc35aad369e7b1284cc48101df60155: Status 404 returned error can't find the container with id 7c9ae346dacbec23f20df450be35156d7dc35aad369e7b1284cc48101df60155 Nov 02 04:04:11.958582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:11.958548 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" event={"ID":"6be50840-13ca-4a52-a9c1-f159af4e9312","Type":"ContainerStarted","Data":"c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898"} Nov 02 04:04:11.959003 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:11.958604 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" event={"ID":"6be50840-13ca-4a52-a9c1-f159af4e9312","Type":"ContainerStarted","Data":"7c9ae346dacbec23f20df450be35156d7dc35aad369e7b1284cc48101df60155"} Nov 02 04:04:11.959003 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:11.958667 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:12.076522 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:12.076471 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" podStartSLOduration=2.0764572709999998 podStartE2EDuration="2.076457271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:12.074656004 +0000 UTC m=+4594.070356947" watchObservedRunningTime="2026-11-02 04:04:12.076457271 +0000 UTC m=+4594.072158214" Nov 02 04:04:12.114736 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:12.114707 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb801bbf-9cd2-489e-985f-cd17ef2638de" path="/var/lib/kubelet/pods/bb801bbf-9cd2-489e-985f-cd17ef2638de/volumes" Nov 02 04:04:13.469056 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:13.469021 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp"] Nov 02 04:04:13.576182 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:13.576152 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb"] Nov 02 04:04:18.206843 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:18.206802 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:04:18.469525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:18.469450 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" containerName="payload-processing" containerID="cri-o://1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb" gracePeriod=25 Nov 02 04:04:18.577527 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:18.577467 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerName="payload-pre-processing" containerID="cri-o://52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c" gracePeriod=25 Nov 02 04:04:21.377214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:21.377170 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:04:21.377629 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:21.377213 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:04:22.237246 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.237182 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 04:04:22.304918 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.304890 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume\") pod \"7a9445ad-44a7-43a1-948a-d14b7363d153\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " Nov 02 04:04:22.305097 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.305045 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55\" (UniqueName: \"kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55\") pod \"7a9445ad-44a7-43a1-948a-d14b7363d153\" (UID: \"7a9445ad-44a7-43a1-948a-d14b7363d153\") " Nov 02 04:04:22.305439 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.305324 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume" pod "7a9445ad-44a7-43a1-948a-d14b7363d153" (UID: "7a9445ad-44a7-43a1-948a-d14b7363d153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:22.307395 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.307372 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55" pod "7a9445ad-44a7-43a1-948a-d14b7363d153" (UID: "7a9445ad-44a7-43a1-948a-d14b7363d153"). InnerVolumeSpecName "kube-api-access-s7m55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:22.406622 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.406589 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9445ad-44a7-43a1-948a-d14b7363d153-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:22.406622 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.406616 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7m55\" (UniqueName: \"kubernetes.io/projected/7a9445ad-44a7-43a1-948a-d14b7363d153-kube-api-access-s7m55\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:22.967577 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:22.967545 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:04:23.029446 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.029419 4868 generic.go:356] "Generic (PLEG): container finished" podID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerID="1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705" exitCode=0 Nov 02 04:04:23.029597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.029460 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" Nov 02 04:04:23.029597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.029504 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" event={"ID":"7a9445ad-44a7-43a1-948a-d14b7363d153","Type":"ContainerDied","Data":"1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705"} Nov 02 04:04:23.029597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.029567 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x" event={"ID":"7a9445ad-44a7-43a1-948a-d14b7363d153","Type":"ContainerDied","Data":"555f55e412299e3b25d2fe21b7e9d622105eaf88679ff1114cedaf412d1652af"} Nov 02 04:04:23.029787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.029600 4868 scope.go:122] "RemoveContainer" containerID="1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705" Nov 02 04:04:23.040021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.039996 4868 scope.go:122] "RemoveContainer" containerID="1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705" Nov 02 04:04:23.040314 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:23.040294 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705\": container with ID starting with 1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705 not found: ID does not exist" containerID="1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705" Nov 02 04:04:23.040425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.040321 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705"} err="failed to get container status \"1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705\": rpc error: code = NotFound desc = could not find container \"1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705\": container with ID starting with 1360b0075513b18746adb876818e7125a71817fbe1a6990069e81ee2ab9ff705 not found: ID does not exist" Nov 02 04:04:23.193842 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.193812 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x"] Nov 02 04:04:23.195792 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:23.195768 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d94cc3-c65c96ff9-hl57x"] Nov 02 04:04:24.114072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.114044 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" path="/var/lib/kubelet/pods/7a9445ad-44a7-43a1-948a-d14b7363d153/volumes" Nov 02 04:04:24.264093 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.264058 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" containerID="cri-o://d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523" gracePeriod=45 Nov 02 04:04:24.266125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.266093 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" containerID="cri-o://500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335" gracePeriod=45 Nov 02 04:04:24.539155 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.539127 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 04:04:24.542303 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.542285 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 04:04:24.627442 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627388 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir\") pod \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " Nov 02 04:04:24.627442 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627420 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm\") pod \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " Nov 02 04:04:24.627442 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627441 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache\") pod \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " Nov 02 04:04:24.627675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627495 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home\") pod \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " Nov 02 04:04:24.627675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627524 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs\" (UniqueName: \"kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs\") pod \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " Nov 02 04:04:24.627675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627577 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs\") pod \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " Nov 02 04:04:24.627675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627623 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm\") pod \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " Nov 02 04:04:24.627675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627660 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd\" (UniqueName: \"kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd\") pod \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " Nov 02 04:04:24.627925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627664 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir" pod "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" (UID: "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.627925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627696 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir\") pod \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " Nov 02 04:04:24.627925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627740 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache\") pod \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " Nov 02 04:04:24.627925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627745 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home" pod "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" (UID: "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.627925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627831 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home\") pod \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\" (UID: \"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd\") " Nov 02 04:04:24.627925 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627896 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n\" (UniqueName: \"kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n\") pod \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\" (UID: \"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413\") " Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627955 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache" pod "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" (UID: "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.627964 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir" pod "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" (UID: "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628181 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home" pod "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" (UID: "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628236 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache" pod "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" (UID: "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628382 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628404 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628418 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628434 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628448 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.628479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.628461 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.630006 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.629982 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm" pod "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" (UID: "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.630224 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.630204 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs" pod "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" (UID: "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:24.630291 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.630221 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm" pod "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" (UID: "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:04:24.630291 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.630256 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd" pod "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" (UID: "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd"). InnerVolumeSpecName "kube-api-access-vrxbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:24.630411 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.630391 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs" pod "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" (UID: "dcd478ed-4edd-4b4f-a11e-8ce86f9321cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:24.630550 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.630528 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n" pod "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" (UID: "58dbe69a-6b18-4d59-8b0a-b0a7c48fb413"). InnerVolumeSpecName "kube-api-access-sst8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:24.729249 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.729220 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.729249 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.729248 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxbd\" (UniqueName: \"kubernetes.io/projected/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-kube-api-access-vrxbd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.729439 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.729260 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sst8n\" (UniqueName: \"kubernetes.io/projected/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-kube-api-access-sst8n\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.729439 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.729269 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.729439 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.729278 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:24.729439 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:24.729289 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:25.043583 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.043507 4868 generic.go:356] "Generic (PLEG): container finished" podID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerID="500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335" exitCode=0 Nov 02 04:04:25.043583 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.043559 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" Nov 02 04:04:25.043789 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.043590 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" event={"ID":"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd","Type":"ContainerDied","Data":"500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335"} Nov 02 04:04:25.043789 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.043633 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg" event={"ID":"dcd478ed-4edd-4b4f-a11e-8ce86f9321cd","Type":"ContainerDied","Data":"f35e8adfb88d4221aee640a7b614636da3321e40ccedcdf1775731201f7736c9"} Nov 02 04:04:25.043789 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.043660 4868 scope.go:122] "RemoveContainer" containerID="500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335" Nov 02 04:04:25.045335 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.045300 4868 generic.go:356] "Generic (PLEG): container finished" podID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerID="d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523" exitCode=0 Nov 02 04:04:25.045473 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.045382 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" event={"ID":"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413","Type":"ContainerDied","Data":"d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523"} Nov 02 04:04:25.045473 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.045423 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" event={"ID":"58dbe69a-6b18-4d59-8b0a-b0a7c48fb413","Type":"ContainerDied","Data":"14ac34c29599d9a43b6ed09a98d447cad67266c23282ea44613921d95ac119e5"} Nov 02 04:04:25.045473 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.045431 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5" Nov 02 04:04:25.053012 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.052995 4868 scope.go:122] "RemoveContainer" containerID="500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335" Nov 02 04:04:25.053303 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:25.053282 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335\": container with ID starting with 500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335 not found: ID does not exist" containerID="500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335" Nov 02 04:04:25.053401 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.053319 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335"} err="failed to get container status \"500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335\": rpc error: code = NotFound desc = could not find container \"500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335\": container with ID starting with 500eb877a3df32106d2dc4057a60bd5e40e6b5e9ced73cec531262a613567335 not found: ID does not exist" Nov 02 04:04:25.053401 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.053342 4868 scope.go:122] "RemoveContainer" containerID="d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523" Nov 02 04:04:25.061990 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.061967 4868 scope.go:122] "RemoveContainer" containerID="d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523" Nov 02 04:04:25.062221 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:25.062206 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523\": container with ID starting with d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523 not found: ID does not exist" containerID="d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523" Nov 02 04:04:25.062273 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.062228 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523"} err="failed to get container status \"d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523\": rpc error: code = NotFound desc = could not find container \"d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523\": container with ID starting with d35173b19ab0503e761ccc20cdddc9773d571991e6d44f973956a8695974e523 not found: ID does not exist" Nov 02 04:04:25.206935 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.206908 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg"] Nov 02 04:04:25.256193 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.256171 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/simulator-w0-1f6e39-kserve-696c494c9c-xbwkg"] Nov 02 04:04:25.325919 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.320447 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5"] Nov 02 04:04:25.325919 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.324325 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f6e39/distinct-w0-1f6e39-kserve-7bb66b65f5-wqxv5"] Nov 02 04:04:25.910029 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.909997 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz"] Nov 02 04:04:25.910465 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910451 4868 cpu_manager.go:414] "containerMap: removing container" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" Nov 02 04:04:25.910529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910466 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" Nov 02 04:04:25.910529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910500 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerName="payload-pre-processing" Nov 02 04:04:25.910529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910506 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerName="payload-pre-processing" Nov 02 04:04:25.910529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910514 4868 cpu_manager.go:414] "containerMap: removing container" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" Nov 02 04:04:25.910529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910520 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" Nov 02 04:04:25.910724 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910609 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9445ad-44a7-43a1-948a-d14b7363d153" containerName="payload-pre-processing" Nov 02 04:04:25.910724 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910619 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" containerName="main" Nov 02 04:04:25.910724 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910627 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="01774e87-b148-4801-b4a8-7bf0564cd846" containerName="istio-proxy" Nov 02 04:04:25.910724 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.910636 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" containerName="main" Nov 02 04:04:25.915136 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:25.915114 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.019360 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.019328 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tk8sg\"" Nov 02 04:04:26.021660 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.021636 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz"] Nov 02 04:04:26.041277 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041250 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041388 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041286 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041450 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041430 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041495 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041476 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041545 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041494 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041635 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041618 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041699 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041677 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041758 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041709 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.041798 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.041779 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfzn\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.114218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.114193 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58dbe69a-6b18-4d59-8b0a-b0a7c48fb413" path="/var/lib/kubelet/pods/58dbe69a-6b18-4d59-8b0a-b0a7c48fb413/volumes" Nov 02 04:04:26.114694 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.114679 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd478ed-4edd-4b4f-a11e-8ce86f9321cd" path="/var/lib/kubelet/pods/dcd478ed-4edd-4b4f-a11e-8ce86f9321cd/volumes" Nov 02 04:04:26.143183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143158 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143189 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143239 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143385 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143291 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143385 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143317 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143405 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143472 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143584 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143505 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143584 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143534 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfzn\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143590 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143650 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143802 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143721 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.143802 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.143778 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.144125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.144104 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.145773 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.145754 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.146032 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.146009 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.200132 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.200077 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.200233 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.200207 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfzn\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mtpnz\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.228085 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.228063 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:26.658789 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.658765 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz"] Nov 02 04:04:26.660084 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:26.660059 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ac9382_bad5_4bd0_9bb0_2dba8ba2206a.slice/crio-b8a4ef92188f0acd6b62ac9baf06b9fe67e474d57e7ec28d41476ed4f5a06aa0 WatchSource:0}: Error finding container b8a4ef92188f0acd6b62ac9baf06b9fe67e474d57e7ec28d41476ed4f5a06aa0: Status 404 returned error can't find the container with id b8a4ef92188f0acd6b62ac9baf06b9fe67e474d57e7ec28d41476ed4f5a06aa0 Nov 02 04:04:26.662629 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.662596 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:04:26.662730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.662673 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:04:26.662730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:26.662721 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:04:27.064414 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:27.064314 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" event={"ID":"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a","Type":"ContainerStarted","Data":"6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566"} Nov 02 04:04:27.064414 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:27.064386 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" event={"ID":"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a","Type":"ContainerStarted","Data":"b8a4ef92188f0acd6b62ac9baf06b9fe67e474d57e7ec28d41476ed4f5a06aa0"} Nov 02 04:04:27.194035 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:27.193853 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podStartSLOduration=2.193830037 podStartE2EDuration="2.193830037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:27.190846396 +0000 UTC m=+4609.186547369" watchObservedRunningTime="2026-11-02 04:04:27.193830037 +0000 UTC m=+4609.189530980" Nov 02 04:04:27.229114 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:27.229074 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:27.230644 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:27.230613 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:27.230788 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:27.230671 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:28.228622 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:28.228593 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:28.228804 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:28.228637 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:29.229253 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:29.229217 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:29.229761 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:29.229276 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:30.229175 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:30.229131 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:30.229410 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:30.229198 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:31.229003 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:31.228965 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:31.229207 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:31.229032 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:31.376699 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:31.376662 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:04:31.377172 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:31.376707 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:04:32.229201 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:32.229166 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:32.229450 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:32.229217 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:33.229403 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:33.229369 4868 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mtpnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 04:04:33.229948 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:33.229416 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 04:04:33.446366 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:33.446312 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:04:34.942061 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:34.942029 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:34.943304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:34.943272 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:34.944524 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:34.944495 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:04:38.735323 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.735266 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 04:04:38.830094 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.830071 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 04:04:38.869378 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.869328 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6\" (UniqueName: \"kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6\") pod \"ea61930e-d4ea-460b-834d-49d0318bf194\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " Nov 02 04:04:38.869536 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.869389 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume\") pod \"ea61930e-d4ea-460b-834d-49d0318bf194\" (UID: \"ea61930e-d4ea-460b-834d-49d0318bf194\") " Nov 02 04:04:38.869849 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.869821 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume" pod "ea61930e-d4ea-460b-834d-49d0318bf194" (UID: "ea61930e-d4ea-460b-834d-49d0318bf194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:38.871341 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.871319 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6" pod "ea61930e-d4ea-460b-834d-49d0318bf194" (UID: "ea61930e-d4ea-460b-834d-49d0318bf194"). InnerVolumeSpecName "kube-api-access-bs8b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:38.970884 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.970849 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume\") pod \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " Nov 02 04:04:38.971048 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.970954 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd\" (UniqueName: \"kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd\") pod \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\" (UID: \"4a17e38b-9c40-4b3b-875c-791d4dfb6762\") " Nov 02 04:04:38.971265 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.971239 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume" pod "4a17e38b-9c40-4b3b-875c-791d4dfb6762" (UID: "4a17e38b-9c40-4b3b-875c-791d4dfb6762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:38.971411 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.971388 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8b6\" (UniqueName: \"kubernetes.io/projected/ea61930e-d4ea-460b-834d-49d0318bf194-kube-api-access-bs8b6\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:38.971503 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.971417 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea61930e-d4ea-460b-834d-49d0318bf194-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:38.971503 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.971432 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a17e38b-9c40-4b3b-875c-791d4dfb6762-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:38.973005 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:38.972983 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd" pod "4a17e38b-9c40-4b3b-875c-791d4dfb6762" (UID: "4a17e38b-9c40-4b3b-875c-791d4dfb6762"). InnerVolumeSpecName "kube-api-access-dw5jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:39.072315 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.072256 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5jd\" (UniqueName: \"kubernetes.io/projected/4a17e38b-9c40-4b3b-875c-791d4dfb6762-kube-api-access-dw5jd\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:39.141797 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.141768 4868 generic.go:356] "Generic (PLEG): container finished" podID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerID="52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c" exitCode=0 Nov 02 04:04:39.141922 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.141804 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" Nov 02 04:04:39.141922 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.141850 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" event={"ID":"4a17e38b-9c40-4b3b-875c-791d4dfb6762","Type":"ContainerDied","Data":"52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c"} Nov 02 04:04:39.141922 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.141898 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb" event={"ID":"4a17e38b-9c40-4b3b-875c-791d4dfb6762","Type":"ContainerDied","Data":"ac9aa5d6325c5d807f7c40dd8bc7e4f70bb3a063a1249ca394f159891a553904"} Nov 02 04:04:39.141922 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.141920 4868 scope.go:122] "RemoveContainer" containerID="52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c" Nov 02 04:04:39.143399 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.143377 4868 generic.go:356] "Generic (PLEG): container finished" podID="ea61930e-d4ea-460b-834d-49d0318bf194" containerID="1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb" exitCode=0 Nov 02 04:04:39.143515 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.143428 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" Nov 02 04:04:39.143515 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.143469 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" event={"ID":"ea61930e-d4ea-460b-834d-49d0318bf194","Type":"ContainerDied","Data":"1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb"} Nov 02 04:04:39.143615 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.143518 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp" event={"ID":"ea61930e-d4ea-460b-834d-49d0318bf194","Type":"ContainerDied","Data":"e20709d12561bea2eb4ccc37331c78afb9730055b6373a5226579c02bab9b399"} Nov 02 04:04:39.153562 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.153544 4868 scope.go:122] "RemoveContainer" containerID="52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c" Nov 02 04:04:39.153812 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:39.153782 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c\": container with ID starting with 52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c not found: ID does not exist" containerID="52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c" Nov 02 04:04:39.153894 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.153823 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c"} err="failed to get container status \"52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c\": rpc error: code = NotFound desc = could not find container \"52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c\": container with ID starting with 52018a82e3adee2b25f2ea6e3e95c84daf266a210241e81314211cab8107170c not found: ID does not exist" Nov 02 04:04:39.153894 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.153845 4868 scope.go:122] "RemoveContainer" containerID="1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb" Nov 02 04:04:39.162416 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.162401 4868 scope.go:122] "RemoveContainer" containerID="1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb" Nov 02 04:04:39.162669 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:04:39.162651 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb\": container with ID starting with 1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb not found: ID does not exist" containerID="1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb" Nov 02 04:04:39.162730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.162678 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb"} err="failed to get container status \"1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb\": rpc error: code = NotFound desc = could not find container \"1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb\": container with ID starting with 1229401f39f0ee1707057bf53b1787e1b1f7a73151b984c6268c17a1e4534acb not found: ID does not exist" Nov 02 04:04:39.305079 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.305048 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb"] Nov 02 04:04:39.312029 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.311426 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ac7ab4-7d5c898ff-bgngb"] Nov 02 04:04:39.424402 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.424379 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp"] Nov 02 04:04:39.436387 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.436364 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ac7ab4-6b59c4f57f-q4kzp"] Nov 02 04:04:39.985419 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.985393 4868 scope.go:122] "RemoveContainer" containerID="b88e6ad84d1f86720ae1e2504b8f44c3b987eb9821c4155336e2dddad4624a2d" Nov 02 04:04:39.994546 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:39.994513 4868 scope.go:122] "RemoveContainer" containerID="686ce55a205631a92e6ed6ba7a1d0a191ca7f361c752c4aefc6550a8e2431152" Nov 02 04:04:40.002793 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:40.002770 4868 scope.go:122] "RemoveContainer" containerID="3f970f400185e3102eac510154c3a3736207217beedec3e8c69f1b0d8c3af35a" Nov 02 04:04:40.013123 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:40.013095 4868 scope.go:122] "RemoveContainer" containerID="c2550d1808147b9e1ed9ad7c72c61de29f8422e2dcabe699023beb132ec442da" Nov 02 04:04:40.113941 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:40.113918 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" path="/var/lib/kubelet/pods/4a17e38b-9c40-4b3b-875c-791d4dfb6762/volumes" Nov 02 04:04:40.114292 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:40.114280 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" path="/var/lib/kubelet/pods/ea61930e-d4ea-460b-834d-49d0318bf194/volumes" Nov 02 04:04:44.331211 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:44.331175 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl"] Nov 02 04:04:44.380522 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:44.380490 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d94cc3-8zxbl"] Nov 02 04:04:46.114838 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:46.114803 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b3afc5-3263-49ab-8502-6f43f8cc8d0c" path="/var/lib/kubelet/pods/c0b3afc5-3263-49ab-8502-6f43f8cc8d0c/volumes" Nov 02 04:04:49.880826 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.880742 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj"] Nov 02 04:04:49.881392 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.881372 4868 cpu_manager.go:414] "containerMap: removing container" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerName="payload-pre-processing" Nov 02 04:04:49.881503 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.881393 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerName="payload-pre-processing" Nov 02 04:04:49.881503 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.881409 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" containerName="payload-processing" Nov 02 04:04:49.881503 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.881416 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" containerName="payload-processing" Nov 02 04:04:49.881665 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.881528 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a17e38b-9c40-4b3b-875c-791d4dfb6762" containerName="payload-pre-processing" Nov 02 04:04:49.881665 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.881544 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea61930e-d4ea-460b-834d-49d0318bf194" containerName="payload-processing" Nov 02 04:04:49.884621 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.884599 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:49.936183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.936154 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc"] Nov 02 04:04:49.940396 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.940339 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj"] Nov 02 04:04:49.940602 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.940455 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:49.983560 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.983535 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:04:49.983719 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.983583 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:04:49.989035 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.989010 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:49.989156 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.989044 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zql4j\" (UniqueName: \"kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:49.989214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.989202 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:49.996827 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:49.996802 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc"] Nov 02 04:04:50.089853 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.089821 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.090003 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.089876 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.090003 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.089909 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.090241 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.090016 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.090241 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.090051 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zql4j\" (UniqueName: \"kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.090241 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.090114 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2br9m\" (UniqueName: \"kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.090738 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.090703 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.092813 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.092793 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.147406 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.147318 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zql4j\" (UniqueName: \"kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j\") pod \"maas-api-revoke-sub-e8a8972b-dxkdj\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.191763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.191737 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.191901 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.191777 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.191901 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.191831 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2br9m\" (UniqueName: \"kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.192500 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.192478 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.193983 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.193961 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.196953 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.196937 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:50.248686 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.248657 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2br9m\" (UniqueName: \"kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m\") pod \"maas-api-revoke-sub-b21d5110-6c7tc\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.256164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.255931 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:50.582689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.582665 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj"] Nov 02 04:04:50.584760 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:50.584725 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215bde34_daec_4af3_baed_4eb1bf4bd191.slice/crio-68c4fdd9f12e2a06685702fb8c43f51f9be5bd963f830f3675059a47dafbd39a WatchSource:0}: Error finding container 68c4fdd9f12e2a06685702fb8c43f51f9be5bd963f830f3675059a47dafbd39a: Status 404 returned error can't find the container with id 68c4fdd9f12e2a06685702fb8c43f51f9be5bd963f830f3675059a47dafbd39a Nov 02 04:04:50.691634 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.691609 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc"] Nov 02 04:04:50.693304 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:50.693273 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c81922_d957_4a89_8b5d_bda90db91028.slice/crio-b6d7fb8385e159577db7e07826325661f7309f59b6033024f5700fde3a84817a WatchSource:0}: Error finding container b6d7fb8385e159577db7e07826325661f7309f59b6033024f5700fde3a84817a: Status 404 returned error can't find the container with id b6d7fb8385e159577db7e07826325661f7309f59b6033024f5700fde3a84817a Nov 02 04:04:50.718211 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.718186 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8"] Nov 02 04:04:50.720607 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:50.720582 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ac7ab4-wp4r8"] Nov 02 04:04:51.239368 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:51.239324 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" event={"ID":"50c81922-d957-4a89-8b5d-bda90db91028","Type":"ContainerStarted","Data":"1de16847d7d185aa30135af81a5cab91b12bf280446194c681d6e02978f2e356"} Nov 02 04:04:51.239785 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:51.239397 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" event={"ID":"50c81922-d957-4a89-8b5d-bda90db91028","Type":"ContainerStarted","Data":"b6d7fb8385e159577db7e07826325661f7309f59b6033024f5700fde3a84817a"} Nov 02 04:04:51.240973 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:51.240949 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" event={"ID":"215bde34-daec-4af3-baed-4eb1bf4bd191","Type":"ContainerStarted","Data":"bedaf8a7f8e830ad7a0235d4c84f4043aa4ea4c12cd66955753dfea5024b1b04"} Nov 02 04:04:51.241084 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:51.240988 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" event={"ID":"215bde34-daec-4af3-baed-4eb1bf4bd191","Type":"ContainerStarted","Data":"68c4fdd9f12e2a06685702fb8c43f51f9be5bd963f830f3675059a47dafbd39a"} Nov 02 04:04:51.354706 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:51.354658 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" podStartSLOduration=2.354645343 podStartE2EDuration="2.354645343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:51.352778253 +0000 UTC m=+4633.348479196" watchObservedRunningTime="2026-11-02 04:04:51.354645343 +0000 UTC m=+4633.350346327" Nov 02 04:04:52.115185 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.115155 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbf7b4b-4e08-4d39-8e1f-982045f55275" path="/var/lib/kubelet/pods/dbbf7b4b-4e08-4d39-8e1f-982045f55275/volumes" Nov 02 04:04:52.248677 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.248645 4868 generic.go:356] "Generic (PLEG): container finished" podID="50c81922-d957-4a89-8b5d-bda90db91028" containerID="1de16847d7d185aa30135af81a5cab91b12bf280446194c681d6e02978f2e356" exitCode=0 Nov 02 04:04:52.249089 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.248676 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" event={"ID":"50c81922-d957-4a89-8b5d-bda90db91028","Type":"ContainerDied","Data":"1de16847d7d185aa30135af81a5cab91b12bf280446194c681d6e02978f2e356"} Nov 02 04:04:52.250382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.250342 4868 generic.go:356] "Generic (PLEG): container finished" podID="215bde34-daec-4af3-baed-4eb1bf4bd191" containerID="bedaf8a7f8e830ad7a0235d4c84f4043aa4ea4c12cd66955753dfea5024b1b04" exitCode=0 Nov 02 04:04:52.250529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.250384 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" event={"ID":"215bde34-daec-4af3-baed-4eb1bf4bd191","Type":"ContainerDied","Data":"bedaf8a7f8e830ad7a0235d4c84f4043aa4ea4c12cd66955753dfea5024b1b04"} Nov 02 04:04:52.363362 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.363313 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" podStartSLOduration=3.363300511 podStartE2EDuration="3.363300511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:51.516777057 +0000 UTC m=+4633.512478021" watchObservedRunningTime="2026-11-02 04:04:52.363300511 +0000 UTC m=+4634.359001669" Nov 02 04:04:52.741969 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.741936 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t"] Nov 02 04:04:52.745642 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.745626 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:52.804326 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.804284 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t"] Nov 02 04:04:52.916368 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.916327 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:52.916558 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.916487 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:52.916558 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:52.916507 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfbr\" (UniqueName: \"kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.017983 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.017893 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.018168 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.017991 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.018168 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.018009 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfbr\" (UniqueName: \"kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.018671 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.018642 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.020392 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.020368 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.074466 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.074437 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfbr\" (UniqueName: \"kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr\") pod \"maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.358942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.358906 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:53.413970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.413940 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:53.417440 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.417417 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:53.522718 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.522692 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m\" (UniqueName: \"kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m\") pod \"50c81922-d957-4a89-8b5d-bda90db91028\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " Nov 02 04:04:53.522841 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.522746 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca\") pod \"215bde34-daec-4af3-baed-4eb1bf4bd191\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " Nov 02 04:04:53.522841 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.522836 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca\") pod \"50c81922-d957-4a89-8b5d-bda90db91028\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " Nov 02 04:04:53.522917 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.522865 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca\") pod \"215bde34-daec-4af3-baed-4eb1bf4bd191\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " Nov 02 04:04:53.522977 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.522956 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca\") pod \"50c81922-d957-4a89-8b5d-bda90db91028\" (UID: \"50c81922-d957-4a89-8b5d-bda90db91028\") " Nov 02 04:04:53.523032 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.523009 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j\" (UniqueName: \"kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j\") pod \"215bde34-daec-4af3-baed-4eb1bf4bd191\" (UID: \"215bde34-daec-4af3-baed-4eb1bf4bd191\") " Nov 02 04:04:53.523411 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.523157 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca" pod "215bde34-daec-4af3-baed-4eb1bf4bd191" (UID: "215bde34-daec-4af3-baed-4eb1bf4bd191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:53.523537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.523410 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca" pod "50c81922-d957-4a89-8b5d-bda90db91028" (UID: "50c81922-d957-4a89-8b5d-bda90db91028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:53.523537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.523434 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:53.524905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.524881 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca" pod "50c81922-d957-4a89-8b5d-bda90db91028" (UID: "50c81922-d957-4a89-8b5d-bda90db91028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:53.524991 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.524883 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m" pod "50c81922-d957-4a89-8b5d-bda90db91028" (UID: "50c81922-d957-4a89-8b5d-bda90db91028"). InnerVolumeSpecName "kube-api-access-2br9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:53.524991 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.524943 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca" pod "215bde34-daec-4af3-baed-4eb1bf4bd191" (UID: "215bde34-daec-4af3-baed-4eb1bf4bd191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:53.525072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.525001 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j" pod "215bde34-daec-4af3-baed-4eb1bf4bd191" (UID: "215bde34-daec-4af3-baed-4eb1bf4bd191"). InnerVolumeSpecName "kube-api-access-zql4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:53.624430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.624402 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:53.624430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.624428 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/215bde34-daec-4af3-baed-4eb1bf4bd191-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:53.624604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.624438 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50c81922-d957-4a89-8b5d-bda90db91028-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:53.624604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.624448 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zql4j\" (UniqueName: \"kubernetes.io/projected/215bde34-daec-4af3-baed-4eb1bf4bd191-kube-api-access-zql4j\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:53.624604 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.624456 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2br9m\" (UniqueName: \"kubernetes.io/projected/50c81922-d957-4a89-8b5d-bda90db91028-kube-api-access-2br9m\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:53.762369 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:53.762331 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t"] Nov 02 04:04:53.764011 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:53.763988 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c623cd_ebf3_464c_a862_b094a6565b8a.slice/crio-66deb815641e4dbfce6b4e4dbca1f9daa64bc3e938e89f1d7422ae96916d2db1 WatchSource:0}: Error finding container 66deb815641e4dbfce6b4e4dbca1f9daa64bc3e938e89f1d7422ae96916d2db1: Status 404 returned error can't find the container with id 66deb815641e4dbfce6b4e4dbca1f9daa64bc3e938e89f1d7422ae96916d2db1 Nov 02 04:04:54.266499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.266429 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" Nov 02 04:04:54.266499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.266437 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc" event={"ID":"50c81922-d957-4a89-8b5d-bda90db91028","Type":"ContainerDied","Data":"b6d7fb8385e159577db7e07826325661f7309f59b6033024f5700fde3a84817a"} Nov 02 04:04:54.266499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.266489 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6d7fb8385e159577db7e07826325661f7309f59b6033024f5700fde3a84817a" Nov 02 04:04:54.267913 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.267896 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" Nov 02 04:04:54.268022 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.267948 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj" event={"ID":"215bde34-daec-4af3-baed-4eb1bf4bd191","Type":"ContainerDied","Data":"68c4fdd9f12e2a06685702fb8c43f51f9be5bd963f830f3675059a47dafbd39a"} Nov 02 04:04:54.268022 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.267993 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c4fdd9f12e2a06685702fb8c43f51f9be5bd963f830f3675059a47dafbd39a" Nov 02 04:04:54.269764 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.269744 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" event={"ID":"57c623cd-ebf3-464c-a862-b094a6565b8a","Type":"ContainerStarted","Data":"b4e6adc2a7517425d6407d7714694ddcde234abe1532946dac8ad8f3926ed88a"} Nov 02 04:04:54.269883 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.269780 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" event={"ID":"57c623cd-ebf3-464c-a862-b094a6565b8a","Type":"ContainerStarted","Data":"66deb815641e4dbfce6b4e4dbca1f9daa64bc3e938e89f1d7422ae96916d2db1"} Nov 02 04:04:54.382912 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:54.382870 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" podStartSLOduration=2.382857587 podStartE2EDuration="2.382857587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:04:54.381630742 +0000 UTC m=+4636.377331686" watchObservedRunningTime="2026-11-02 04:04:54.382857587 +0000 UTC m=+4636.378558521" Nov 02 04:04:55.277470 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:55.277431 4868 generic.go:356] "Generic (PLEG): container finished" podID="57c623cd-ebf3-464c-a862-b094a6565b8a" containerID="b4e6adc2a7517425d6407d7714694ddcde234abe1532946dac8ad8f3926ed88a" exitCode=0 Nov 02 04:04:55.277685 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:55.277518 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" event={"ID":"57c623cd-ebf3-464c-a862-b094a6565b8a","Type":"ContainerDied","Data":"b4e6adc2a7517425d6407d7714694ddcde234abe1532946dac8ad8f3926ed88a"} Nov 02 04:04:56.434103 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.434080 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:56.555730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.555669 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca\") pod \"57c623cd-ebf3-464c-a862-b094a6565b8a\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " Nov 02 04:04:56.555860 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.555732 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca\") pod \"57c623cd-ebf3-464c-a862-b094a6565b8a\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " Nov 02 04:04:56.555860 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.555849 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr\" (UniqueName: \"kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr\") pod \"57c623cd-ebf3-464c-a862-b094a6565b8a\" (UID: \"57c623cd-ebf3-464c-a862-b094a6565b8a\") " Nov 02 04:04:56.556101 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.556073 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca" pod "57c623cd-ebf3-464c-a862-b094a6565b8a" (UID: "57c623cd-ebf3-464c-a862-b094a6565b8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:04:56.556258 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.556241 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:56.558057 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.558022 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr" pod "57c623cd-ebf3-464c-a862-b094a6565b8a" (UID: "57c623cd-ebf3-464c-a862-b094a6565b8a"). InnerVolumeSpecName "kube-api-access-shfbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:04:56.558477 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.558448 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca" pod "57c623cd-ebf3-464c-a862-b094a6565b8a" (UID: "57c623cd-ebf3-464c-a862-b094a6565b8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:04:56.657447 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.657421 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfbr\" (UniqueName: \"kubernetes.io/projected/57c623cd-ebf3-464c-a862-b094a6565b8a-kube-api-access-shfbr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:56.657569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:56.657452 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57c623cd-ebf3-464c-a862-b094a6565b8a-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:04:57.187653 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:57.187622 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp"] Nov 02 04:04:57.196721 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:57.196692 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x"] Nov 02 04:04:57.291873 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:57.291843 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" Nov 02 04:04:57.291873 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:57.291867 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t" event={"ID":"57c623cd-ebf3-464c-a862-b094a6565b8a","Type":"ContainerDied","Data":"66deb815641e4dbfce6b4e4dbca1f9daa64bc3e938e89f1d7422ae96916d2db1"} Nov 02 04:04:57.292078 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:57.291918 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66deb815641e4dbfce6b4e4dbca1f9daa64bc3e938e89f1d7422ae96916d2db1" Nov 02 04:04:58.004992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.004955 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw"] Nov 02 04:04:58.005540 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.005195 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" containerID="cri-o://33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15" gracePeriod=30 Nov 02 04:04:58.959992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.959959 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq"] Nov 02 04:04:58.960479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960460 4868 cpu_manager.go:414] "containerMap: removing container" podUID="57c623cd-ebf3-464c-a862-b094a6565b8a" containerName="revoke-keys" Nov 02 04:04:58.960590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960480 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c623cd-ebf3-464c-a862-b094a6565b8a" containerName="revoke-keys" Nov 02 04:04:58.960590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960505 4868 cpu_manager.go:414] "containerMap: removing container" podUID="50c81922-d957-4a89-8b5d-bda90db91028" containerName="revoke-keys" Nov 02 04:04:58.960590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960512 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c81922-d957-4a89-8b5d-bda90db91028" containerName="revoke-keys" Nov 02 04:04:58.960590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960534 4868 cpu_manager.go:414] "containerMap: removing container" podUID="215bde34-daec-4af3-baed-4eb1bf4bd191" containerName="revoke-keys" Nov 02 04:04:58.960590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960543 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215bde34-daec-4af3-baed-4eb1bf4bd191" containerName="revoke-keys" Nov 02 04:04:58.960850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960645 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c623cd-ebf3-464c-a862-b094a6565b8a" containerName="revoke-keys" Nov 02 04:04:58.960850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960662 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c81922-d957-4a89-8b5d-bda90db91028" containerName="revoke-keys" Nov 02 04:04:58.960850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.960676 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="215bde34-daec-4af3-baed-4eb1bf4bd191" containerName="revoke-keys" Nov 02 04:04:58.964859 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:58.964837 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.020205 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.020175 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq"] Nov 02 04:04:59.066548 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.066516 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:04:59.066548 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.066542 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:04:59.078833 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.078802 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghgq\" (UniqueName: \"kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.078960 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.078849 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.079040 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.078970 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.180197 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.180165 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghgq\" (UniqueName: \"kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.180376 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.180216 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.180376 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.180273 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.180983 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.180959 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.182599 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.182578 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.238493 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.238424 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghgq\" (UniqueName: \"kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq\") pod \"maas-api-revoke-keys-e2e-worker-w4-398828-94mlq\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.279580 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.279549 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:04:59.651614 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:04:59.651586 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq"] Nov 02 04:04:59.652755 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:04:59.652723 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc78d07e_3458_47f0_9ed0_9dd8cc818504.slice/crio-3c75b5766e33a69007e068670e382617e35aa0a483ffe96d4ccfa99e173f2cf9 WatchSource:0}: Error finding container 3c75b5766e33a69007e068670e382617e35aa0a483ffe96d4ccfa99e173f2cf9: Status 404 returned error can't find the container with id 3c75b5766e33a69007e068670e382617e35aa0a483ffe96d4ccfa99e173f2cf9 Nov 02 04:05:00.312000 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:00.311915 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" event={"ID":"fc78d07e-3458-47f0-9ed0-9dd8cc818504","Type":"ContainerStarted","Data":"da43ecc0a160f5d397ac3159c30ee78908f60eccedde519641fa56338f063e50"} Nov 02 04:05:00.312000 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:00.311975 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" event={"ID":"fc78d07e-3458-47f0-9ed0-9dd8cc818504","Type":"ContainerStarted","Data":"3c75b5766e33a69007e068670e382617e35aa0a483ffe96d4ccfa99e173f2cf9"} Nov 02 04:05:00.426815 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:00.426765 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" podStartSLOduration=2.426750841 podStartE2EDuration="2.426750841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:05:00.424932903 +0000 UTC m=+4642.420633845" watchObservedRunningTime="2026-11-02 04:05:00.426750841 +0000 UTC m=+4642.422451783" Nov 02 04:05:01.322910 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:01.322862 4868 generic.go:356] "Generic (PLEG): container finished" podID="fc78d07e-3458-47f0-9ed0-9dd8cc818504" containerID="da43ecc0a160f5d397ac3159c30ee78908f60eccedde519641fa56338f063e50" exitCode=0 Nov 02 04:05:01.323308 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:01.322944 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" event={"ID":"fc78d07e-3458-47f0-9ed0-9dd8cc818504","Type":"ContainerDied","Data":"da43ecc0a160f5d397ac3159c30ee78908f60eccedde519641fa56338f063e50"} Nov 02 04:05:02.465321 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.465296 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:05:02.512783 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.512757 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca\") pod \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " Nov 02 04:05:02.512926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.512797 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq\" (UniqueName: \"kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq\") pod \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " Nov 02 04:05:02.512926 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.512817 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca\") pod \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\" (UID: \"fc78d07e-3458-47f0-9ed0-9dd8cc818504\") " Nov 02 04:05:02.513242 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.513217 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca" pod "fc78d07e-3458-47f0-9ed0-9dd8cc818504" (UID: "fc78d07e-3458-47f0-9ed0-9dd8cc818504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:02.514832 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.514811 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca" pod "fc78d07e-3458-47f0-9ed0-9dd8cc818504" (UID: "fc78d07e-3458-47f0-9ed0-9dd8cc818504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:02.514935 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.514913 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq" pod "fc78d07e-3458-47f0-9ed0-9dd8cc818504" (UID: "fc78d07e-3458-47f0-9ed0-9dd8cc818504"). InnerVolumeSpecName "kube-api-access-tghgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:02.614537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.614502 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:02.614537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.614529 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghgq\" (UniqueName: \"kubernetes.io/projected/fc78d07e-3458-47f0-9ed0-9dd8cc818504-kube-api-access-tghgq\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:02.614537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:02.614541 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78d07e-3458-47f0-9ed0-9dd8cc818504-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.040567 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.040534 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz"] Nov 02 04:05:03.042159 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.042130 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3b27a5-vkdvz"] Nov 02 04:05:03.244457 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.244432 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 04:05:03.324054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.323963 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324008 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324050 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324095 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324130 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324177 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324247 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324300 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324647 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324371 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert\") pod \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\" (UID: \"c7b1e1b5-0896-480f-8b77-e1d853f16f09\") " Nov 02 04:05:03.324647 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324403 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:03.324647 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324502 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:03.324647 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324522 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:03.324905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324665 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:03.324905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324835 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:03.324905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324867 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.324905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324898 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.325052 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324913 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.325052 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.324927 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.326245 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.326219 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "kube-api-access-wdc29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:03.326479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.326459 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:03.326565 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.326550 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:03.326680 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.326659 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo" pod "c7b1e1b5-0896-480f-8b77-e1d853f16f09" (UID: "c7b1e1b5-0896-480f-8b77-e1d853f16f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:05:03.335156 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.335134 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" Nov 02 04:05:03.335156 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.335148 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq" event={"ID":"fc78d07e-3458-47f0-9ed0-9dd8cc818504","Type":"ContainerDied","Data":"3c75b5766e33a69007e068670e382617e35aa0a483ffe96d4ccfa99e173f2cf9"} Nov 02 04:05:03.335316 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.335197 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c75b5766e33a69007e068670e382617e35aa0a483ffe96d4ccfa99e173f2cf9" Nov 02 04:05:03.336950 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.336930 4868 generic.go:356] "Generic (PLEG): container finished" podID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerID="33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15" exitCode=0 Nov 02 04:05:03.337048 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.336970 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" Nov 02 04:05:03.337048 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.337017 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" event={"ID":"c7b1e1b5-0896-480f-8b77-e1d853f16f09","Type":"ContainerDied","Data":"33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15"} Nov 02 04:05:03.337130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.337081 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw" event={"ID":"c7b1e1b5-0896-480f-8b77-e1d853f16f09","Type":"ContainerDied","Data":"e335017ee155615e9862b8ce6bc4ef83f2aa7ea5aebc3239ff27e6d6de6af962"} Nov 02 04:05:03.337130 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.337112 4868 scope.go:122] "RemoveContainer" containerID="33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15" Nov 02 04:05:03.348866 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.348852 4868 scope.go:122] "RemoveContainer" containerID="51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8" Nov 02 04:05:03.357127 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.357107 4868 scope.go:122] "RemoveContainer" containerID="33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15" Nov 02 04:05:03.357406 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:03.357387 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15\": container with ID starting with 33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15 not found: ID does not exist" containerID="33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15" Nov 02 04:05:03.357457 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.357414 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15"} err="failed to get container status \"33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15\": rpc error: code = NotFound desc = could not find container \"33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15\": container with ID starting with 33767c0c16ecaec38f41c4271fcfb091b6ea4db130245e1848a6e3cd93b67b15 not found: ID does not exist" Nov 02 04:05:03.357457 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.357432 4868 scope.go:122] "RemoveContainer" containerID="51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8" Nov 02 04:05:03.357664 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:03.357647 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8\": container with ID starting with 51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8 not found: ID does not exist" containerID="51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8" Nov 02 04:05:03.357710 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.357670 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8"} err="failed to get container status \"51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8\": rpc error: code = NotFound desc = could not find container \"51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8\": container with ID starting with 51d78fcbd04a8d9801e965a3c1528467b99da383c04d6eac3189d356c8f7dcc8 not found: ID does not exist" Nov 02 04:05:03.426425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.426393 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdc29\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-kube-api-access-wdc29\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.426425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.426424 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.426592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.426441 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.426592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.426456 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.426592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.426471 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b1e1b5-0896-480f-8b77-e1d853f16f09-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:03.446771 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:03.446737 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:05:03.456092 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.456065 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h"] Nov 02 04:05:03.473787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.473763 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw"] Nov 02 04:05:03.479091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:03.479070 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ac7ab4-openshift-default-66dcc65c6f-bq7hw"] Nov 02 04:05:04.113814 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:04.113784 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125eb568-0bc5-431b-a994-dcadb7eaa3de" path="/var/lib/kubelet/pods/125eb568-0bc5-431b-a994-dcadb7eaa3de/volumes" Nov 02 04:05:04.114248 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:04.114232 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" path="/var/lib/kubelet/pods/c7b1e1b5-0896-480f-8b77-e1d853f16f09/volumes" Nov 02 04:05:07.557120 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557090 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f"] Nov 02 04:05:07.557544 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557534 4868 cpu_manager.go:414] "containerMap: removing container" podUID="fc78d07e-3458-47f0-9ed0-9dd8cc818504" containerName="revoke-keys" Nov 02 04:05:07.557591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557545 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc78d07e-3458-47f0-9ed0-9dd8cc818504" containerName="revoke-keys" Nov 02 04:05:07.557591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557558 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" Nov 02 04:05:07.557591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557563 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" Nov 02 04:05:07.557591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557574 4868 cpu_manager.go:414] "containerMap: removing container" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" Nov 02 04:05:07.557591 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557579 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" Nov 02 04:05:07.557751 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557636 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc78d07e-3458-47f0-9ed0-9dd8cc818504" containerName="revoke-keys" Nov 02 04:05:07.557751 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557644 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" Nov 02 04:05:07.557751 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.557650 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b1e1b5-0896-480f-8b77-e1d853f16f09" containerName="istio-proxy" Nov 02 04:05:07.562254 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.562236 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.612713 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.612692 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb"] Nov 02 04:05:07.616240 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.616214 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.660545 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.660520 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:05:07.660545 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.660548 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:05:07.662488 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.662461 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f"] Nov 02 04:05:07.665626 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.665606 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xff\" (UniqueName: \"kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.665736 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.665645 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.665736 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.665672 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.665814 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.665793 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.665848 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.665836 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.665963 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.665946 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvkq\" (UniqueName: \"kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.681319 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.681296 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb"] Nov 02 04:05:07.766758 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.766726 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvkq\" (UniqueName: \"kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.766912 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.766801 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xff\" (UniqueName: \"kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.766912 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.766840 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.766912 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.766875 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.767125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.766911 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.767125 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.766956 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.767637 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.767609 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.767772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.767639 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.769194 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.769168 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.769299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.769280 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.823277 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.823227 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvkq\" (UniqueName: \"kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq\") pod \"maas-api-revoke-sub-9237d21c-wvswb\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:07.823277 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.823217 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xff\" (UniqueName: \"kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff\") pod \"maas-api-revoke-sub-9fe48b66-qjm8f\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.875493 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.875469 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:07.930525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:07.930498 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:08.246732 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:08.246709 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f"] Nov 02 04:05:08.248061 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:05:08.248027 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a5aaeb_0d21_4c1c_9a47_343b011b2bb8.slice/crio-09cd6ab9474d1e14f7c3a902866c7c625d7f131c989fe9be3c6657f3457407ab WatchSource:0}: Error finding container 09cd6ab9474d1e14f7c3a902866c7c625d7f131c989fe9be3c6657f3457407ab: Status 404 returned error can't find the container with id 09cd6ab9474d1e14f7c3a902866c7c625d7f131c989fe9be3c6657f3457407ab Nov 02 04:05:08.355226 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:08.355202 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb"] Nov 02 04:05:08.356100 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:05:08.356069 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d013783_f450_41cd_ae4e_30d0b366233f.slice/crio-99a7b3787d0744383d619300dac2637c8381a3ab7bcc97a82d41649bcb8ffa34 WatchSource:0}: Error finding container 99a7b3787d0744383d619300dac2637c8381a3ab7bcc97a82d41649bcb8ffa34: Status 404 returned error can't find the container with id 99a7b3787d0744383d619300dac2637c8381a3ab7bcc97a82d41649bcb8ffa34 Nov 02 04:05:08.367701 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:08.367674 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" event={"ID":"7d013783-f450-41cd-ae4e-30d0b366233f","Type":"ContainerStarted","Data":"99a7b3787d0744383d619300dac2637c8381a3ab7bcc97a82d41649bcb8ffa34"} Nov 02 04:05:08.369059 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:08.369036 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" event={"ID":"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8","Type":"ContainerStarted","Data":"09cd6ab9474d1e14f7c3a902866c7c625d7f131c989fe9be3c6657f3457407ab"} Nov 02 04:05:09.377876 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:09.377840 4868 generic.go:356] "Generic (PLEG): container finished" podID="20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" containerID="4e12e64ea7e2bbaf3b1f1ee724b18759bd8931d93d4a869545164143e9aca0af" exitCode=0 Nov 02 04:05:09.378265 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:09.377926 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" event={"ID":"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8","Type":"ContainerDied","Data":"4e12e64ea7e2bbaf3b1f1ee724b18759bd8931d93d4a869545164143e9aca0af"} Nov 02 04:05:09.379618 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:09.379592 4868 generic.go:356] "Generic (PLEG): container finished" podID="7d013783-f450-41cd-ae4e-30d0b366233f" containerID="7fa2a43a52570a32ef2d0761dbe4c344ee0132d8726ced40e89229408f799dea" exitCode=0 Nov 02 04:05:09.379749 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:09.379630 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" event={"ID":"7d013783-f450-41cd-ae4e-30d0b366233f","Type":"ContainerDied","Data":"7fa2a43a52570a32ef2d0761dbe4c344ee0132d8726ced40e89229408f799dea"} Nov 02 04:05:10.544218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.544195 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:10.546366 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.546335 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:10.596689 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.596660 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff\" (UniqueName: \"kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff\") pod \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " Nov 02 04:05:10.596850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.596740 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca\") pod \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " Nov 02 04:05:10.596850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.596771 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca\") pod \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\" (UID: \"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8\") " Nov 02 04:05:10.596850 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.596817 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca\") pod \"7d013783-f450-41cd-ae4e-30d0b366233f\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " Nov 02 04:05:10.597011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.596856 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq\" (UniqueName: \"kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq\") pod \"7d013783-f450-41cd-ae4e-30d0b366233f\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " Nov 02 04:05:10.597011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.596915 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca\") pod \"7d013783-f450-41cd-ae4e-30d0b366233f\" (UID: \"7d013783-f450-41cd-ae4e-30d0b366233f\") " Nov 02 04:05:10.597194 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.597161 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca" pod "20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" (UID: "20a5aaeb-0d21-4c1c-9a47-343b011b2bb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:10.597393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.597344 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:10.597961 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.597929 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca" pod "7d013783-f450-41cd-ae4e-30d0b366233f" (UID: "7d013783-f450-41cd-ae4e-30d0b366233f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:10.599267 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.599237 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca" pod "20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" (UID: "20a5aaeb-0d21-4c1c-9a47-343b011b2bb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:10.599479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.599460 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff" pod "20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" (UID: "20a5aaeb-0d21-4c1c-9a47-343b011b2bb8"). InnerVolumeSpecName "kube-api-access-x2xff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:10.599532 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.599481 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca" pod "7d013783-f450-41cd-ae4e-30d0b366233f" (UID: "7d013783-f450-41cd-ae4e-30d0b366233f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:10.599532 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.599501 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq" pod "7d013783-f450-41cd-ae4e-30d0b366233f" (UID: "7d013783-f450-41cd-ae4e-30d0b366233f"). InnerVolumeSpecName "kube-api-access-hnvkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:10.699393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.699299 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:10.699393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.699328 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:10.699393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.699338 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvkq\" (UniqueName: \"kubernetes.io/projected/7d013783-f450-41cd-ae4e-30d0b366233f-kube-api-access-hnvkq\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:10.699393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.699376 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d013783-f450-41cd-ae4e-30d0b366233f-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:10.699393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.699389 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xff\" (UniqueName: \"kubernetes.io/projected/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8-kube-api-access-x2xff\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:10.723997 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.723966 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65"] Nov 02 04:05:10.724381 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.724368 4868 cpu_manager.go:414] "containerMap: removing container" podUID="7d013783-f450-41cd-ae4e-30d0b366233f" containerName="revoke-keys" Nov 02 04:05:10.724432 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.724389 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d013783-f450-41cd-ae4e-30d0b366233f" containerName="revoke-keys" Nov 02 04:05:10.724432 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.724421 4868 cpu_manager.go:414] "containerMap: removing container" podUID="20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" containerName="revoke-keys" Nov 02 04:05:10.724432 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.724428 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" containerName="revoke-keys" Nov 02 04:05:10.724529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.724493 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d013783-f450-41cd-ae4e-30d0b366233f" containerName="revoke-keys" Nov 02 04:05:10.724529 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.724504 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" containerName="revoke-keys" Nov 02 04:05:10.727851 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.727833 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.785518 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.785486 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65"] Nov 02 04:05:10.800378 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.800331 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgd8b\" (UniqueName: \"kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.800616 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.800594 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.800702 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.800634 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.914943 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.914905 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.914943 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.914957 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.914943 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.914988 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgd8b\" (UniqueName: \"kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.915839 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.915789 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.918870 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.918838 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:10.972401 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:10.972331 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgd8b\" (UniqueName: \"kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b\") pod \"maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:11.042282 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.042247 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:11.400701 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.400662 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" Nov 02 04:05:11.400701 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.400688 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb" event={"ID":"7d013783-f450-41cd-ae4e-30d0b366233f","Type":"ContainerDied","Data":"99a7b3787d0744383d619300dac2637c8381a3ab7bcc97a82d41649bcb8ffa34"} Nov 02 04:05:11.400948 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.400773 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a7b3787d0744383d619300dac2637c8381a3ab7bcc97a82d41649bcb8ffa34" Nov 02 04:05:11.402321 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.402296 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" Nov 02 04:05:11.402321 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.402303 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f" event={"ID":"20a5aaeb-0d21-4c1c-9a47-343b011b2bb8","Type":"ContainerDied","Data":"09cd6ab9474d1e14f7c3a902866c7c625d7f131c989fe9be3c6657f3457407ab"} Nov 02 04:05:11.402503 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.402336 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09cd6ab9474d1e14f7c3a902866c7c625d7f131c989fe9be3c6657f3457407ab" Nov 02 04:05:11.414402 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:11.414368 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65"] Nov 02 04:05:11.415817 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:05:11.415796 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d526ca_ddd2_4844_9018_02dedacee118.slice/crio-fa90c94957e28e7db2c9ba077b68ef5c2733be092782ff45cdce5f352b890404 WatchSource:0}: Error finding container fa90c94957e28e7db2c9ba077b68ef5c2733be092782ff45cdce5f352b890404: Status 404 returned error can't find the container with id fa90c94957e28e7db2c9ba077b68ef5c2733be092782ff45cdce5f352b890404 Nov 02 04:05:12.208016 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.207960 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" containerID="cri-o://ad2114f9b898c38439048478db51110f80bbd2316699daaacb7d3e91fef505d0" gracePeriod=45 Nov 02 04:05:12.222776 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.221713 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" containerID="cri-o://c6092239ba80cfc2d6715b135311a22462c2e8999e30ae2ca4e733d2e926b1f2" gracePeriod=45 Nov 02 04:05:12.416800 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.415786 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" event={"ID":"51d526ca-ddd2-4844-9018-02dedacee118","Type":"ContainerStarted","Data":"16a9a7e4d0ce9d1b202666f193e437a756dc647a1bb81601923be578df4dd9e4"} Nov 02 04:05:12.416800 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.415851 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" event={"ID":"51d526ca-ddd2-4844-9018-02dedacee118","Type":"ContainerStarted","Data":"fa90c94957e28e7db2c9ba077b68ef5c2733be092782ff45cdce5f352b890404"} Nov 02 04:05:12.419418 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.419133 4868 generic.go:356] "Generic (PLEG): container finished" podID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerID="c6092239ba80cfc2d6715b135311a22462c2e8999e30ae2ca4e733d2e926b1f2" exitCode=0 Nov 02 04:05:12.419418 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.419189 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" event={"ID":"1e4f2d56-ebda-4544-9c44-063ebd385386","Type":"ContainerDied","Data":"c6092239ba80cfc2d6715b135311a22462c2e8999e30ae2ca4e733d2e926b1f2"} Nov 02 04:05:12.422102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.422010 4868 generic.go:356] "Generic (PLEG): container finished" podID="6abbaca8-8870-4569-ae4d-4e188d620122" containerID="ad2114f9b898c38439048478db51110f80bbd2316699daaacb7d3e91fef505d0" exitCode=0 Nov 02 04:05:12.422102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.422062 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" event={"ID":"6abbaca8-8870-4569-ae4d-4e188d620122","Type":"ContainerDied","Data":"ad2114f9b898c38439048478db51110f80bbd2316699daaacb7d3e91fef505d0"} Nov 02 04:05:12.495796 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.495775 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 04:05:12.524877 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.524855 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 04:05:12.530211 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.530173 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" podStartSLOduration=2.530161077 podStartE2EDuration="2.530161077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:05:12.528286833 +0000 UTC m=+4654.523987778" watchObservedRunningTime="2026-11-02 04:05:12.530161077 +0000 UTC m=+4654.525862020" Nov 02 04:05:12.535300 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535280 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home\") pod \"6abbaca8-8870-4569-ae4d-4e188d620122\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " Nov 02 04:05:12.535447 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535318 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc\" (UniqueName: \"kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc\") pod \"6abbaca8-8870-4569-ae4d-4e188d620122\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " Nov 02 04:05:12.535447 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535377 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache\") pod \"6abbaca8-8870-4569-ae4d-4e188d620122\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " Nov 02 04:05:12.535574 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535495 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs\" (UniqueName: \"kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs\") pod \"6abbaca8-8870-4569-ae4d-4e188d620122\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " Nov 02 04:05:12.535574 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535526 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm\") pod \"6abbaca8-8870-4569-ae4d-4e188d620122\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " Nov 02 04:05:12.535684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535562 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home" pod "6abbaca8-8870-4569-ae4d-4e188d620122" (UID: "6abbaca8-8870-4569-ae4d-4e188d620122"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.535684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535579 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir\") pod \"6abbaca8-8870-4569-ae4d-4e188d620122\" (UID: \"6abbaca8-8870-4569-ae4d-4e188d620122\") " Nov 02 04:05:12.535684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535645 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache" pod "6abbaca8-8870-4569-ae4d-4e188d620122" (UID: "6abbaca8-8870-4569-ae4d-4e188d620122"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.535922 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.535895 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir" pod "6abbaca8-8870-4569-ae4d-4e188d620122" (UID: "6abbaca8-8870-4569-ae4d-4e188d620122"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.536140 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.536121 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.536248 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.536227 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.537499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.537479 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm" pod "6abbaca8-8870-4569-ae4d-4e188d620122" (UID: "6abbaca8-8870-4569-ae4d-4e188d620122"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.537877 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.537854 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc" pod "6abbaca8-8870-4569-ae4d-4e188d620122" (UID: "6abbaca8-8870-4569-ae4d-4e188d620122"). InnerVolumeSpecName "kube-api-access-jhrhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:12.537877 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.537852 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs" pod "6abbaca8-8870-4569-ae4d-4e188d620122" (UID: "6abbaca8-8870-4569-ae4d-4e188d620122"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:12.636725 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.636695 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs\") pod \"1e4f2d56-ebda-4544-9c44-063ebd385386\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " Nov 02 04:05:12.636879 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.636734 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir\") pod \"1e4f2d56-ebda-4544-9c44-063ebd385386\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " Nov 02 04:05:12.636879 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.636793 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm\") pod \"1e4f2d56-ebda-4544-9c44-063ebd385386\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " Nov 02 04:05:12.636879 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.636854 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home\") pod \"1e4f2d56-ebda-4544-9c44-063ebd385386\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " Nov 02 04:05:12.637121 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.636932 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache\") pod \"1e4f2d56-ebda-4544-9c44-063ebd385386\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " Nov 02 04:05:12.637121 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.636971 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98\" (UniqueName: \"kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98\") pod \"1e4f2d56-ebda-4544-9c44-063ebd385386\" (UID: \"1e4f2d56-ebda-4544-9c44-063ebd385386\") " Nov 02 04:05:12.637121 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637043 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir" pod "1e4f2d56-ebda-4544-9c44-063ebd385386" (UID: "1e4f2d56-ebda-4544-9c44-063ebd385386"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.637299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637260 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6abbaca8-8870-4569-ae4d-4e188d620122-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.637299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637273 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.637299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637282 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6abbaca8-8870-4569-ae4d-4e188d620122-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.637299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637290 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.637299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637300 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrhc\" (UniqueName: \"kubernetes.io/projected/6abbaca8-8870-4569-ae4d-4e188d620122-kube-api-access-jhrhc\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.637596 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637554 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home" pod "1e4f2d56-ebda-4544-9c44-063ebd385386" (UID: "1e4f2d56-ebda-4544-9c44-063ebd385386"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.637596 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.637571 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache" pod "1e4f2d56-ebda-4544-9c44-063ebd385386" (UID: "1e4f2d56-ebda-4544-9c44-063ebd385386"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.639115 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.639088 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs" pod "1e4f2d56-ebda-4544-9c44-063ebd385386" (UID: "1e4f2d56-ebda-4544-9c44-063ebd385386"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:12.639360 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.639324 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm" pod "1e4f2d56-ebda-4544-9c44-063ebd385386" (UID: "1e4f2d56-ebda-4544-9c44-063ebd385386"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:12.639894 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.639875 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98" pod "1e4f2d56-ebda-4544-9c44-063ebd385386" (UID: "1e4f2d56-ebda-4544-9c44-063ebd385386"). InnerVolumeSpecName "kube-api-access-smm98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:12.738430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.738346 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.738430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.738383 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.738430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.738392 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4f2d56-ebda-4544-9c44-063ebd385386-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.738430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.738401 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smm98\" (UniqueName: \"kubernetes.io/projected/1e4f2d56-ebda-4544-9c44-063ebd385386-kube-api-access-smm98\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:12.738430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:12.738411 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4f2d56-ebda-4544-9c44-063ebd385386-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:13.429942 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.429912 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" Nov 02 04:05:13.430475 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.429938 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp" event={"ID":"6abbaca8-8870-4569-ae4d-4e188d620122","Type":"ContainerDied","Data":"11b74989f33267b36bf870f4e785b140e46fd55d80e8f5ef52ef49ba77a411ce"} Nov 02 04:05:13.430475 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.429990 4868 scope.go:122] "RemoveContainer" containerID="ad2114f9b898c38439048478db51110f80bbd2316699daaacb7d3e91fef505d0" Nov 02 04:05:13.431678 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.431654 4868 generic.go:356] "Generic (PLEG): container finished" podID="51d526ca-ddd2-4844-9018-02dedacee118" containerID="16a9a7e4d0ce9d1b202666f193e437a756dc647a1bb81601923be578df4dd9e4" exitCode=0 Nov 02 04:05:13.431829 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.431801 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" event={"ID":"51d526ca-ddd2-4844-9018-02dedacee118","Type":"ContainerDied","Data":"16a9a7e4d0ce9d1b202666f193e437a756dc647a1bb81601923be578df4dd9e4"} Nov 02 04:05:13.433192 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.433174 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" Nov 02 04:05:13.433310 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.433191 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x" event={"ID":"1e4f2d56-ebda-4544-9c44-063ebd385386","Type":"ContainerDied","Data":"e17f12a7bd19b61e5328e1130d2682785aa0e2f462afa7f77e6a669c3517bd34"} Nov 02 04:05:13.440612 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.440597 4868 scope.go:122] "RemoveContainer" containerID="c6092239ba80cfc2d6715b135311a22462c2e8999e30ae2ca4e733d2e926b1f2" Nov 02 04:05:13.606567 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.606018 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x"] Nov 02 04:05:13.609478 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.609446 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/premium-w2-ca6546-kserve-6844f9d785-qtk5x"] Nov 02 04:05:13.714874 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.714808 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp"] Nov 02 04:05:13.718076 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:13.718055 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ca6546/distinct-w2-ca6546-kserve-898b79df6-hkfsp"] Nov 02 04:05:14.114843 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.114816 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" path="/var/lib/kubelet/pods/1e4f2d56-ebda-4544-9c44-063ebd385386/volumes" Nov 02 04:05:14.115266 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.115249 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" path="/var/lib/kubelet/pods/6abbaca8-8870-4569-ae4d-4e188d620122/volumes" Nov 02 04:05:14.573898 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.573874 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:14.653923 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.653901 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b\" (UniqueName: \"kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b\") pod \"51d526ca-ddd2-4844-9018-02dedacee118\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " Nov 02 04:05:14.654054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.653932 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca\") pod \"51d526ca-ddd2-4844-9018-02dedacee118\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " Nov 02 04:05:14.654111 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.654064 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca\") pod \"51d526ca-ddd2-4844-9018-02dedacee118\" (UID: \"51d526ca-ddd2-4844-9018-02dedacee118\") " Nov 02 04:05:14.654436 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.654413 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca" pod "51d526ca-ddd2-4844-9018-02dedacee118" (UID: "51d526ca-ddd2-4844-9018-02dedacee118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:14.655900 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.655880 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b" pod "51d526ca-ddd2-4844-9018-02dedacee118" (UID: "51d526ca-ddd2-4844-9018-02dedacee118"). InnerVolumeSpecName "kube-api-access-kgd8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:14.655969 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.655927 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca" pod "51d526ca-ddd2-4844-9018-02dedacee118" (UID: "51d526ca-ddd2-4844-9018-02dedacee118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:14.755037 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.754964 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:14.755037 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.754991 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgd8b\" (UniqueName: \"kubernetes.io/projected/51d526ca-ddd2-4844-9018-02dedacee118-kube-api-access-kgd8b\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:14.755037 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:14.755003 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d526ca-ddd2-4844-9018-02dedacee118-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:15.139411 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:15.139374 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r"] Nov 02 04:05:15.453385 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:15.453270 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" Nov 02 04:05:15.453385 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:15.453306 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65" event={"ID":"51d526ca-ddd2-4844-9018-02dedacee118","Type":"ContainerDied","Data":"fa90c94957e28e7db2c9ba077b68ef5c2733be092782ff45cdce5f352b890404"} Nov 02 04:05:15.453631 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:15.453427 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa90c94957e28e7db2c9ba077b68ef5c2733be092782ff45cdce5f352b890404" Nov 02 04:05:18.247687 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.247652 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq"] Nov 02 04:05:18.248088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248049 4868 cpu_manager.go:414] "containerMap: removing container" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" Nov 02 04:05:18.248088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248059 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" Nov 02 04:05:18.248088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248072 4868 cpu_manager.go:414] "containerMap: removing container" podUID="51d526ca-ddd2-4844-9018-02dedacee118" containerName="revoke-keys" Nov 02 04:05:18.248088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248077 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d526ca-ddd2-4844-9018-02dedacee118" containerName="revoke-keys" Nov 02 04:05:18.248222 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248094 4868 cpu_manager.go:414] "containerMap: removing container" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" Nov 02 04:05:18.248222 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248099 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" Nov 02 04:05:18.248222 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248176 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4f2d56-ebda-4544-9c44-063ebd385386" containerName="main" Nov 02 04:05:18.248222 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248183 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d526ca-ddd2-4844-9018-02dedacee118" containerName="revoke-keys" Nov 02 04:05:18.248222 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.248192 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abbaca8-8870-4569-ae4d-4e188d620122" containerName="main" Nov 02 04:05:18.251231 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.251212 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.354836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.354807 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:05:18.355010 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.354949 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:05:18.357116 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.357091 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs"] Nov 02 04:05:18.360722 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.360704 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq"] Nov 02 04:05:18.360845 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.360808 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.385718 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.385696 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.385839 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.385745 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.385923 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.385897 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kp52\" (UniqueName: \"kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.406437 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.406408 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs"] Nov 02 04:05:18.475958 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.475928 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" containerID="cri-o://13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497" gracePeriod=45 Nov 02 04:05:18.486660 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.486637 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6hts\" (UniqueName: \"kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.486768 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.486689 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.486768 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.486753 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.486869 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.486856 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.486913 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.486902 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kp52\" (UniqueName: \"kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.486955 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.486939 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.487513 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.487492 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.489407 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.489265 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.543574 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.543555 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kp52\" (UniqueName: \"kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52\") pod \"maas-api-revoke-sub-1ee7e627-9zttq\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.564599 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.564580 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:18.587881 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.587854 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hts\" (UniqueName: \"kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.587987 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.587934 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.588063 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.588010 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.588787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.588760 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.590469 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.590444 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.645142 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.645117 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6hts\" (UniqueName: \"kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts\") pod \"maas-api-revoke-sub-949a867c-dmdfs\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.676739 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.676673 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:18.735911 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.735889 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 04:05:18.790029 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.789958 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9\" (UniqueName: \"kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9\") pod \"71d6712e-e0c6-4897-9c03-65546a869c0a\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " Nov 02 04:05:18.790029 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790014 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir\") pod \"71d6712e-e0c6-4897-9c03-65546a869c0a\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " Nov 02 04:05:18.790205 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790041 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs\") pod \"71d6712e-e0c6-4897-9c03-65546a869c0a\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " Nov 02 04:05:18.790205 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790100 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache\") pod \"71d6712e-e0c6-4897-9c03-65546a869c0a\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " Nov 02 04:05:18.790205 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790169 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home\") pod \"71d6712e-e0c6-4897-9c03-65546a869c0a\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " Nov 02 04:05:18.790323 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790209 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm\") pod \"71d6712e-e0c6-4897-9c03-65546a869c0a\" (UID: \"71d6712e-e0c6-4897-9c03-65546a869c0a\") " Nov 02 04:05:18.790323 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790293 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir" pod "71d6712e-e0c6-4897-9c03-65546a869c0a" (UID: "71d6712e-e0c6-4897-9c03-65546a869c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:18.790449 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790419 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache" pod "71d6712e-e0c6-4897-9c03-65546a869c0a" (UID: "71d6712e-e0c6-4897-9c03-65546a869c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:18.790513 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790458 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home" pod "71d6712e-e0c6-4897-9c03-65546a869c0a" (UID: "71d6712e-e0c6-4897-9c03-65546a869c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:18.790613 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790594 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:18.790702 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790615 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:18.790702 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.790624 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:18.792280 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.792255 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9" pod "71d6712e-e0c6-4897-9c03-65546a869c0a" (UID: "71d6712e-e0c6-4897-9c03-65546a869c0a"). InnerVolumeSpecName "kube-api-access-b4dr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:18.792409 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.792283 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs" pod "71d6712e-e0c6-4897-9c03-65546a869c0a" (UID: "71d6712e-e0c6-4897-9c03-65546a869c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:18.792409 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.792289 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm" pod "71d6712e-e0c6-4897-9c03-65546a869c0a" (UID: "71d6712e-e0c6-4897-9c03-65546a869c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:18.891367 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.891321 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71d6712e-e0c6-4897-9c03-65546a869c0a-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:18.891507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.891376 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dr9\" (UniqueName: \"kubernetes.io/projected/71d6712e-e0c6-4897-9c03-65546a869c0a-kube-api-access-b4dr9\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:18.891507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.891390 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d6712e-e0c6-4897-9c03-65546a869c0a-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:18.963537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:18.963512 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq"] Nov 02 04:05:18.964649 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:05:18.964623 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524fa02f_4485_4aea_92c4_aa703da40e07.slice/crio-d1b4840d7a23d603e514d3fb074db15232bba23320050bb7bdbdfeac4d482e97 WatchSource:0}: Error finding container d1b4840d7a23d603e514d3fb074db15232bba23320050bb7bdbdfeac4d482e97: Status 404 returned error can't find the container with id d1b4840d7a23d603e514d3fb074db15232bba23320050bb7bdbdfeac4d482e97 Nov 02 04:05:19.050122 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.049921 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs"] Nov 02 04:05:19.053154 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:05:19.053124 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42cc639_7f8c_49ef_a88f_0d2d32e7c985.slice/crio-654f23604e6b800d7758bd689dce2602411d023b25207a95eb2a9fdd885ea1b6 WatchSource:0}: Error finding container 654f23604e6b800d7758bd689dce2602411d023b25207a95eb2a9fdd885ea1b6: Status 404 returned error can't find the container with id 654f23604e6b800d7758bd689dce2602411d023b25207a95eb2a9fdd885ea1b6 Nov 02 04:05:19.486484 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.484849 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" event={"ID":"f42cc639-7f8c-49ef-a88f-0d2d32e7c985","Type":"ContainerStarted","Data":"3cb11fcf47c7f2ea1c1c32a9792a23fe616ff19892c0e9cedb0f8fde98f3e215"} Nov 02 04:05:19.486484 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.484922 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" event={"ID":"f42cc639-7f8c-49ef-a88f-0d2d32e7c985","Type":"ContainerStarted","Data":"654f23604e6b800d7758bd689dce2602411d023b25207a95eb2a9fdd885ea1b6"} Nov 02 04:05:19.487167 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.486705 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" event={"ID":"524fa02f-4485-4aea-92c4-aa703da40e07","Type":"ContainerStarted","Data":"1113712ab7737f0d2bf9b0bc116e292427cff36ac739bdf7507ec252679c400a"} Nov 02 04:05:19.487167 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.486757 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" event={"ID":"524fa02f-4485-4aea-92c4-aa703da40e07","Type":"ContainerStarted","Data":"d1b4840d7a23d603e514d3fb074db15232bba23320050bb7bdbdfeac4d482e97"} Nov 02 04:05:19.488285 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.488256 4868 generic.go:356] "Generic (PLEG): container finished" podID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerID="13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497" exitCode=0 Nov 02 04:05:19.488438 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.488285 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" Nov 02 04:05:19.488438 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.488337 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" event={"ID":"71d6712e-e0c6-4897-9c03-65546a869c0a","Type":"ContainerDied","Data":"13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497"} Nov 02 04:05:19.488438 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.488401 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" event={"ID":"71d6712e-e0c6-4897-9c03-65546a869c0a","Type":"ContainerDied","Data":"51dd5d32872a45b5f56f5bed6c7f1ee3795189f49ff6eb8bc9c88ec6cb61737a"} Nov 02 04:05:19.488438 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.488421 4868 scope.go:122] "RemoveContainer" containerID="13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497" Nov 02 04:05:19.507897 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.507879 4868 scope.go:122] "RemoveContainer" containerID="13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497" Nov 02 04:05:19.508158 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:19.508135 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497\": container with ID starting with 13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497 not found: ID does not exist" containerID="13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497" Nov 02 04:05:19.508202 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.508161 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497"} err="failed to get container status \"13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497\": rpc error: code = NotFound desc = could not find container \"13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497\": container with ID starting with 13c9ae7477b04442934853c86873ff45714dfc718300e6b4d00fc6abd237b497 not found: ID does not exist" Nov 02 04:05:19.601977 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.601938 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" podStartSLOduration=1.601926342 podStartE2EDuration="1.601926342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:05:19.600721481 +0000 UTC m=+4661.596422428" watchObservedRunningTime="2026-11-02 04:05:19.601926342 +0000 UTC m=+4661.597627284" Nov 02 04:05:19.682750 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.682716 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 04:05:19.711973 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.711950 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h"] Nov 02 04:05:19.715752 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.715732 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-398828/distinct-w4-398828-kserve-6574c7d75-zhx8h"] Nov 02 04:05:19.830612 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:19.827756 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" podStartSLOduration=1.827740221 podStartE2EDuration="1.827740221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:05:19.825468294 +0000 UTC m=+4661.821169240" watchObservedRunningTime="2026-11-02 04:05:19.827740221 +0000 UTC m=+4661.823441168" Nov 02 04:05:20.002042 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:20.002012 4868 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42cc639_7f8c_49ef_a88f_0d2d32e7c985.slice/crio-conmon-3cb11fcf47c7f2ea1c1c32a9792a23fe616ff19892c0e9cedb0f8fde98f3e215.scope\": RecentStats: unable to find data in memory cache]" Nov 02 04:05:20.112742 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:20.112713 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" path="/var/lib/kubelet/pods/71d6712e-e0c6-4897-9c03-65546a869c0a/volumes" Nov 02 04:05:20.495277 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:20.495243 4868 generic.go:356] "Generic (PLEG): container finished" podID="f42cc639-7f8c-49ef-a88f-0d2d32e7c985" containerID="3cb11fcf47c7f2ea1c1c32a9792a23fe616ff19892c0e9cedb0f8fde98f3e215" exitCode=0 Nov 02 04:05:20.495789 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:20.495327 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" event={"ID":"f42cc639-7f8c-49ef-a88f-0d2d32e7c985","Type":"ContainerDied","Data":"3cb11fcf47c7f2ea1c1c32a9792a23fe616ff19892c0e9cedb0f8fde98f3e215"} Nov 02 04:05:20.496937 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:20.496912 4868 generic.go:356] "Generic (PLEG): container finished" podID="524fa02f-4485-4aea-92c4-aa703da40e07" containerID="1113712ab7737f0d2bf9b0bc116e292427cff36ac739bdf7507ec252679c400a" exitCode=0 Nov 02 04:05:20.497063 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:20.496995 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" event={"ID":"524fa02f-4485-4aea-92c4-aa703da40e07","Type":"ContainerDied","Data":"1113712ab7737f0d2bf9b0bc116e292427cff36ac739bdf7507ec252679c400a"} Nov 02 04:05:21.707840 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.707820 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:21.709978 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.709962 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:21.816304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816243 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca\") pod \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " Nov 02 04:05:21.816304 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816274 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca\") pod \"524fa02f-4485-4aea-92c4-aa703da40e07\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " Nov 02 04:05:21.816523 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816319 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca\") pod \"524fa02f-4485-4aea-92c4-aa703da40e07\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " Nov 02 04:05:21.816523 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816480 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts\" (UniqueName: \"kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts\") pod \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " Nov 02 04:05:21.816644 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816534 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52\" (UniqueName: \"kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52\") pod \"524fa02f-4485-4aea-92c4-aa703da40e07\" (UID: \"524fa02f-4485-4aea-92c4-aa703da40e07\") " Nov 02 04:05:21.816644 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816564 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca\") pod \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\" (UID: \"f42cc639-7f8c-49ef-a88f-0d2d32e7c985\") " Nov 02 04:05:21.816851 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.816826 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca" pod "524fa02f-4485-4aea-92c4-aa703da40e07" (UID: "524fa02f-4485-4aea-92c4-aa703da40e07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:21.817046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.817015 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca" pod "f42cc639-7f8c-49ef-a88f-0d2d32e7c985" (UID: "f42cc639-7f8c-49ef-a88f-0d2d32e7c985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:21.817171 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.817074 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:21.818520 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.818495 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca" pod "f42cc639-7f8c-49ef-a88f-0d2d32e7c985" (UID: "f42cc639-7f8c-49ef-a88f-0d2d32e7c985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:21.818730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.818710 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52" pod "524fa02f-4485-4aea-92c4-aa703da40e07" (UID: "524fa02f-4485-4aea-92c4-aa703da40e07"). InnerVolumeSpecName "kube-api-access-7kp52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:21.818791 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.818745 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts" pod "f42cc639-7f8c-49ef-a88f-0d2d32e7c985" (UID: "f42cc639-7f8c-49ef-a88f-0d2d32e7c985"). InnerVolumeSpecName "kube-api-access-n6hts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:21.818882 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.818869 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca" pod "524fa02f-4485-4aea-92c4-aa703da40e07" (UID: "524fa02f-4485-4aea-92c4-aa703da40e07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:21.917745 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.917712 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:21.917745 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.917743 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/524fa02f-4485-4aea-92c4-aa703da40e07-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:21.917881 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.917753 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6hts\" (UniqueName: \"kubernetes.io/projected/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-kube-api-access-n6hts\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:21.917881 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.917763 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kp52\" (UniqueName: \"kubernetes.io/projected/524fa02f-4485-4aea-92c4-aa703da40e07-kube-api-access-7kp52\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:21.917881 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:21.917772 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42cc639-7f8c-49ef-a88f-0d2d32e7c985-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:22.511061 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:22.510968 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" Nov 02 04:05:22.511225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:22.510963 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs" event={"ID":"f42cc639-7f8c-49ef-a88f-0d2d32e7c985","Type":"ContainerDied","Data":"654f23604e6b800d7758bd689dce2602411d023b25207a95eb2a9fdd885ea1b6"} Nov 02 04:05:22.511225 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:22.511120 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654f23604e6b800d7758bd689dce2602411d023b25207a95eb2a9fdd885ea1b6" Nov 02 04:05:22.512496 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:22.512475 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" Nov 02 04:05:22.512618 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:22.512515 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq" event={"ID":"524fa02f-4485-4aea-92c4-aa703da40e07","Type":"ContainerDied","Data":"d1b4840d7a23d603e514d3fb074db15232bba23320050bb7bdbdfeac4d482e97"} Nov 02 04:05:22.512618 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:22.512564 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b4840d7a23d603e514d3fb074db15232bba23320050bb7bdbdfeac4d482e97" Nov 02 04:05:25.403193 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:25.403156 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2"] Nov 02 04:05:25.458988 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:25.458958 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1f6e39-7c6z2"] Nov 02 04:05:26.113995 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:26.113959 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088baf26-62c0-4fec-968f-084f1a2723e5" path="/var/lib/kubelet/pods/088baf26-62c0-4fec-968f-084f1a2723e5/volumes" Nov 02 04:05:30.158871 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.158831 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" containerID="cri-o://41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd" gracePeriod=45 Nov 02 04:05:30.436057 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.436028 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 04:05:30.540578 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540546 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs\" (UniqueName: \"kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs\") pod \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " Nov 02 04:05:30.540578 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540579 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache\") pod \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " Nov 02 04:05:30.540772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540611 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw\" (UniqueName: \"kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw\") pod \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " Nov 02 04:05:30.540772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540734 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir\") pod \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " Nov 02 04:05:30.540772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540761 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm\") pod \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " Nov 02 04:05:30.540972 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540813 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home\") pod \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\" (UID: \"62a4763e-0808-463d-9a43-0b7e20ff7aa7\") " Nov 02 04:05:30.540972 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.540891 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache" pod "62a4763e-0808-463d-9a43-0b7e20ff7aa7" (UID: "62a4763e-0808-463d-9a43-0b7e20ff7aa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:30.541082 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.541039 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir" pod "62a4763e-0808-463d-9a43-0b7e20ff7aa7" (UID: "62a4763e-0808-463d-9a43-0b7e20ff7aa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:30.541218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.541195 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home" pod "62a4763e-0808-463d-9a43-0b7e20ff7aa7" (UID: "62a4763e-0808-463d-9a43-0b7e20ff7aa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:30.541309 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.541237 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:30.541309 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.541262 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:30.543115 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.543083 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm" pod "62a4763e-0808-463d-9a43-0b7e20ff7aa7" (UID: "62a4763e-0808-463d-9a43-0b7e20ff7aa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:05:30.543115 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.543093 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs" pod "62a4763e-0808-463d-9a43-0b7e20ff7aa7" (UID: "62a4763e-0808-463d-9a43-0b7e20ff7aa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:30.543278 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.543178 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw" pod "62a4763e-0808-463d-9a43-0b7e20ff7aa7" (UID: "62a4763e-0808-463d-9a43-0b7e20ff7aa7"). InnerVolumeSpecName "kube-api-access-6wczw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:30.584858 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.584831 4868 generic.go:356] "Generic (PLEG): container finished" podID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerID="41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd" exitCode=0 Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.584896 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.584926 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" event={"ID":"62a4763e-0808-463d-9a43-0b7e20ff7aa7","Type":"ContainerDied","Data":"41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd"} Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.584970 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r" event={"ID":"62a4763e-0808-463d-9a43-0b7e20ff7aa7","Type":"ContainerDied","Data":"512617c59f70e5f08038b7d3ed894e4b2732e1e4b27259dffec1b756c28b3428"} Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.584989 4868 scope.go:122] "RemoveContainer" containerID="41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd" Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.594243 4868 scope.go:122] "RemoveContainer" containerID="41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd" Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:30.594526 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd\": container with ID starting with 41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd not found: ID does not exist" containerID="41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd" Nov 02 04:05:30.599592 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.594557 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd"} err="failed to get container status \"41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd\": rpc error: code = NotFound desc = could not find container \"41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd\": container with ID starting with 41570731e0253f0a0872ff0fe4d7ccd7b7bc3911092312efc15b8a80eee66fcd not found: ID does not exist" Nov 02 04:05:30.641828 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.641802 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:30.641828 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.641830 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a4763e-0808-463d-9a43-0b7e20ff7aa7-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:30.642018 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.641844 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a4763e-0808-463d-9a43-0b7e20ff7aa7-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:30.642018 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.641858 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wczw\" (UniqueName: \"kubernetes.io/projected/62a4763e-0808-463d-9a43-0b7e20ff7aa7-kube-api-access-6wczw\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:30.749015 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.748977 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r"] Nov 02 04:05:30.753118 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:30.753092 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5ac93f/distinct-2-w1-5ac93f-kserve-57ddf65dd8-kbt5r"] Nov 02 04:05:31.593385 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:31.593302 4868 generic.go:356] "Generic (PLEG): container finished" podID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerID="6dbc89d926e8146143175e592a9128962fa902c7c499db1dbd72857b0843bf63" exitCode=0 Nov 02 04:05:31.593759 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:31.593386 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" event={"ID":"e2693bf6-5c99-45ed-bed7-605704705d8f","Type":"ContainerDied","Data":"6dbc89d926e8146143175e592a9128962fa902c7c499db1dbd72857b0843bf63"} Nov 02 04:05:31.593806 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:31.593763 4868 scope.go:122] "RemoveContainer" containerID="6dbc89d926e8146143175e592a9128962fa902c7c499db1dbd72857b0843bf63" Nov 02 04:05:31.594969 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:31.594943 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:05:31.595040 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:31.595017 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:05:31.595088 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:31.595050 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:05:32.113509 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:32.113469 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" path="/var/lib/kubelet/pods/62a4763e-0808-463d-9a43-0b7e20ff7aa7/volumes" Nov 02 04:05:32.601311 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:32.601278 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" event={"ID":"e2693bf6-5c99-45ed-bed7-605704705d8f","Type":"ContainerStarted","Data":"d7bf238f0f2ce9f9064db3f520c6ef8b9d7c3fb7bec34f196c779cdeb7755f81"} Nov 02 04:05:33.284320 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:33.284288 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:05:33.285284 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:33.285264 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 04:05:33.285340 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:33.285307 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 04:05:33.468306 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:05:33.468273 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:05:34.284458 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:34.284418 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 04:05:34.284867 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:34.284484 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 04:05:34.891499 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:34.891465 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:05:35.284051 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:35.283950 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 04:05:35.284051 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:35.284010 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 04:05:36.285000 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:36.284959 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 04:05:36.285463 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:36.285018 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 04:05:37.284220 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:37.284182 4868 patch_prober.go:29] interesting pod/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 02 04:05:37.284430 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:37.284244 4868 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 02 04:05:38.288072 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:38.288034 4868 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:05:38.288924 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:38.288905 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:05:42.793216 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:42.793182 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t"] Nov 02 04:05:42.842425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:42.842395 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ca6546-f4p2t"] Nov 02 04:05:44.114955 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:44.114923 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c623cd-ebf3-464c-a862-b094a6565b8a" path="/var/lib/kubelet/pods/57c623cd-ebf3-464c-a862-b094a6565b8a/volumes" Nov 02 04:05:51.860474 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.860434 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj"] Nov 02 04:05:51.861074 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861040 4868 cpu_manager.go:414] "containerMap: removing container" podUID="524fa02f-4485-4aea-92c4-aa703da40e07" containerName="revoke-keys" Nov 02 04:05:51.861074 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861061 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="524fa02f-4485-4aea-92c4-aa703da40e07" containerName="revoke-keys" Nov 02 04:05:51.861213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861083 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f42cc639-7f8c-49ef-a88f-0d2d32e7c985" containerName="revoke-keys" Nov 02 04:05:51.861213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861091 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42cc639-7f8c-49ef-a88f-0d2d32e7c985" containerName="revoke-keys" Nov 02 04:05:51.861213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861107 4868 cpu_manager.go:414] "containerMap: removing container" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" Nov 02 04:05:51.861213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861116 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" Nov 02 04:05:51.861213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861131 4868 cpu_manager.go:414] "containerMap: removing container" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" Nov 02 04:05:51.861213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861139 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" Nov 02 04:05:51.861569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861250 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="524fa02f-4485-4aea-92c4-aa703da40e07" containerName="revoke-keys" Nov 02 04:05:51.861569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861267 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42cc639-7f8c-49ef-a88f-0d2d32e7c985" containerName="revoke-keys" Nov 02 04:05:51.861569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861277 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a4763e-0808-463d-9a43-0b7e20ff7aa7" containerName="main" Nov 02 04:05:51.861569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.861286 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d6712e-e0c6-4897-9c03-65546a869c0a" containerName="main" Nov 02 04:05:51.864603 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.864580 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:51.918010 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.917962 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 04:05:51.920513 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.920487 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9z5bk\"" Nov 02 04:05:51.921071 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.921053 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj"] Nov 02 04:05:51.953020 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.952995 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:51.953133 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.953038 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7sr\" (UniqueName: \"kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:51.953187 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:51.953136 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.054556 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.054526 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.054740 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.054571 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7sr\" (UniqueName: \"kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.054740 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.054709 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.055306 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.055286 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.056968 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.056939 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.114024 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.113960 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7sr\" (UniqueName: \"kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr\") pod \"maas-api-revoke-sub-6dbf92f1-j9ttj\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.176952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.176914 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:52.567403 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.567019 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj"] Nov 02 04:05:52.724742 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:52.724707 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" event={"ID":"172e64fa-d2c6-4dbc-b571-80fb94dc221c","Type":"ContainerStarted","Data":"71ad30b4fef776150e63d8e2a55a73d8731817775538bcc393b9a28459e055c4"} Nov 02 04:05:53.732607 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:53.732564 4868 generic.go:356] "Generic (PLEG): container finished" podID="172e64fa-d2c6-4dbc-b571-80fb94dc221c" containerID="91791b4d8757fb2eb88099b72876da95b7c40015e0cb4d20b942f8581f45b260" exitCode=0 Nov 02 04:05:53.733023 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:53.732608 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" event={"ID":"172e64fa-d2c6-4dbc-b571-80fb94dc221c","Type":"ContainerDied","Data":"91791b4d8757fb2eb88099b72876da95b7c40015e0cb4d20b942f8581f45b260"} Nov 02 04:05:54.125337 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:54.125302 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq"] Nov 02 04:05:54.133489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:54.133464 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-398828-94mlq"] Nov 02 04:05:54.872517 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:54.872488 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:55.001930 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.001864 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca\") pod \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " Nov 02 04:05:55.002064 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.001932 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr\" (UniqueName: \"kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr\") pod \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " Nov 02 04:05:55.002064 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.001975 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca\") pod \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\" (UID: \"172e64fa-d2c6-4dbc-b571-80fb94dc221c\") " Nov 02 04:05:55.002387 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.002336 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca" pod "172e64fa-d2c6-4dbc-b571-80fb94dc221c" (UID: "172e64fa-d2c6-4dbc-b571-80fb94dc221c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:05:55.004128 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.004099 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca" pod "172e64fa-d2c6-4dbc-b571-80fb94dc221c" (UID: "172e64fa-d2c6-4dbc-b571-80fb94dc221c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:05:55.004759 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.004415 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr" pod "172e64fa-d2c6-4dbc-b571-80fb94dc221c" (UID: "172e64fa-d2c6-4dbc-b571-80fb94dc221c"). InnerVolumeSpecName "kube-api-access-gn7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:05:55.103009 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.102972 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7sr\" (UniqueName: \"kubernetes.io/projected/172e64fa-d2c6-4dbc-b571-80fb94dc221c-kube-api-access-gn7sr\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:55.103009 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.103004 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-openshift-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:55.103213 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.103020 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172e64fa-d2c6-4dbc-b571-80fb94dc221c-maas-api-xks-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:05:55.746838 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.746811 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" Nov 02 04:05:55.746838 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.746821 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj" event={"ID":"172e64fa-d2c6-4dbc-b571-80fb94dc221c","Type":"ContainerDied","Data":"71ad30b4fef776150e63d8e2a55a73d8731817775538bcc393b9a28459e055c4"} Nov 02 04:05:55.747166 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:55.746868 4868 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ad30b4fef776150e63d8e2a55a73d8731817775538bcc393b9a28459e055c4" Nov 02 04:05:56.113617 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:05:56.113586 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc78d07e-3458-47f0-9ed0-9dd8cc818504" path="/var/lib/kubelet/pods/fc78d07e-3458-47f0-9ed0-9dd8cc818504/volumes" Nov 02 04:06:00.179001 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.178970 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t"] Nov 02 04:06:00.179505 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.179362 4868 cpu_manager.go:414] "containerMap: removing container" podUID="172e64fa-d2c6-4dbc-b571-80fb94dc221c" containerName="revoke-keys" Nov 02 04:06:00.179505 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.179375 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172e64fa-d2c6-4dbc-b571-80fb94dc221c" containerName="revoke-keys" Nov 02 04:06:00.179505 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.179461 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="172e64fa-d2c6-4dbc-b571-80fb94dc221c" containerName="revoke-keys" Nov 02 04:06:00.183605 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.183585 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.235556 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.235536 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-398828-openshift-default-dockercfg-dqq8t\"" Nov 02 04:06:00.287572 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.287540 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t"] Nov 02 04:06:00.349630 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349598 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.349780 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349651 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.349857 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349785 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.349857 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349845 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.349953 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349889 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.349953 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349942 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.350045 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349964 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586x8\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.350045 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.349996 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.350045 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.350011 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.450889 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.450828 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.450889 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.450872 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451081 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.450899 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451081 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.450915 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451081 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451031 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451081 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451076 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586x8\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451313 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451128 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451313 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451151 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451313 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451215 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451313 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451270 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451313 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451214 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451547 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451335 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451614 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451581 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.451854 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.451830 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.453450 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.453426 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.453547 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.453533 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.507597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.507567 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.508122 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.508100 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586x8\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8\") pod \"e2e-worker-w4-398828-openshift-default-5647988965-84v2t\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:00.796121 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:00.796047 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:01.049775 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.049662 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t"] Nov 02 04:06:01.229247 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.229221 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t"] Nov 02 04:06:01.230227 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:06:01.230187 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad66f25e_461a_4c36_bcaf_4dc59ad1f59d.slice/crio-d030d6d8c82da118b6a7ecff600c644292bd29fd844bac7d8e01860f1f0807d7 WatchSource:0}: Error finding container d030d6d8c82da118b6a7ecff600c644292bd29fd844bac7d8e01860f1f0807d7: Status 404 returned error can't find the container with id d030d6d8c82da118b6a7ecff600c644292bd29fd844bac7d8e01860f1f0807d7 Nov 02 04:06:01.232586 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.232556 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:06:01.232687 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.232616 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:06:01.232687 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.232644 4868 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 04:06:01.784746 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.784702 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" event={"ID":"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d","Type":"ContainerStarted","Data":"5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d"} Nov 02 04:06:01.784970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.784745 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" podUID="ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" containerName="istio-proxy" containerID="cri-o://5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d" gracePeriod=30 Nov 02 04:06:01.784970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.784764 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" event={"ID":"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d","Type":"ContainerStarted","Data":"d030d6d8c82da118b6a7ecff600c644292bd29fd844bac7d8e01860f1f0807d7"} Nov 02 04:06:01.902302 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:01.902248 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" podStartSLOduration=1.9022330429999998 podStartE2EDuration="1.902233043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 04:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:06:01.901055671 +0000 UTC m=+4703.896756628" watchObservedRunningTime="2026-11-02 04:06:01.902233043 +0000 UTC m=+4703.897933982" Nov 02 04:06:03.445066 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:06:03.445026 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:06:07.025434 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.025406 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:07.217164 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217135 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217183 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217231 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217261 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217279 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217382 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217328 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217697 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217390 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217697 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217449 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217697 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217482 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy\") pod \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\" (UID: \"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d\") " Nov 02 04:06:07.217868 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217735 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:07.217868 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217779 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:07.217868 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217818 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:07.218038 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.217956 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.218038 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.218006 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.218038 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.218021 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.218209 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.218068 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:07.218209 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.218157 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:06:07.219554 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.219529 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:06:07.219710 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.219690 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "kube-api-access-586x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:06:07.219834 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.219808 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:07.220393 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.220375 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token" pod "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" (UID: "ad66f25e-461a-4c36-bcaf-4dc59ad1f59d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:06:07.319362 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.319332 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.319362 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.319366 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.319525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.319375 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586x8\" (UniqueName: \"kubernetes.io/projected/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-kube-api-access-586x8\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.319525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.319386 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.319525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.319394 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.319525 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.319404 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:07.824112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.824082 4868 generic.go:356] "Generic (PLEG): container finished" podID="ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" containerID="5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d" exitCode=0 Nov 02 04:06:07.824294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.824135 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" Nov 02 04:06:07.824294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.824170 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" event={"ID":"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d","Type":"ContainerDied","Data":"5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d"} Nov 02 04:06:07.824294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.824232 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t" event={"ID":"ad66f25e-461a-4c36-bcaf-4dc59ad1f59d","Type":"ContainerDied","Data":"d030d6d8c82da118b6a7ecff600c644292bd29fd844bac7d8e01860f1f0807d7"} Nov 02 04:06:07.824294 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.824265 4868 scope.go:122] "RemoveContainer" containerID="5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d" Nov 02 04:06:07.834495 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.834478 4868 scope.go:122] "RemoveContainer" containerID="5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d" Nov 02 04:06:07.834754 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:06:07.834737 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d\": container with ID starting with 5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d not found: ID does not exist" containerID="5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d" Nov 02 04:06:07.834810 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:07.834761 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d"} err="failed to get container status \"5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d\": rpc error: code = NotFound desc = could not find container \"5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d\": container with ID starting with 5a0bbc928bfff824905f2ff04ba95a5a484008f6a57bd3e4754f7720f3979e4d not found: ID does not exist" Nov 02 04:06:08.039511 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:08.039477 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t"] Nov 02 04:06:08.044381 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:08.044321 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-398828-openshift-default-5647988965-84v2t"] Nov 02 04:06:08.114888 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:08.114859 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" path="/var/lib/kubelet/pods/ad66f25e-461a-4c36-bcaf-4dc59ad1f59d/volumes" Nov 02 04:06:24.628835 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:24.628756 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65"] Nov 02 04:06:24.677879 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:24.677846 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5ac93f-jwq65"] Nov 02 04:06:26.113426 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:26.113396 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d526ca-ddd2-4844-9018-02dedacee118" path="/var/lib/kubelet/pods/51d526ca-ddd2-4844-9018-02dedacee118/volumes" Nov 02 04:06:31.247801 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:31.247768 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2"] Nov 02 04:06:31.248255 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:31.247995 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" containerID="cri-o://4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4" gracePeriod=30 Nov 02 04:06:33.445808 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:06:33.445762 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:06:36.499764 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.499738 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 04:06:36.583635 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583571 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.583635 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583605 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.583635 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583626 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.583878 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583684 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.583878 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583750 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.583878 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583778 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.583878 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583865 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.584108 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583895 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.584108 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583949 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token\") pod \"83b3ebbb-20a1-442c-8ed7-7c930929d773\" (UID: \"83b3ebbb-20a1-442c-8ed7-7c930929d773\") " Nov 02 04:06:36.584108 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583880 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:36.584108 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.583952 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:36.584108 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.584064 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:36.584428 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.584332 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.584428 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.584378 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.584428 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.584395 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.584428 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.584416 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:06:36.584669 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.584488 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:36.585830 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.585805 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "kube-api-access-9d2qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:06:36.586011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.585994 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:06:36.586011 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.585999 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:06:36.586135 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.586117 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy" pod "83b3ebbb-20a1-442c-8ed7-7c930929d773" (UID: "83b3ebbb-20a1-442c-8ed7-7c930929d773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:06:36.685839 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.685812 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.685839 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.685837 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b3ebbb-20a1-442c-8ed7-7c930929d773-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.686024 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.685847 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b3ebbb-20a1-442c-8ed7-7c930929d773-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.686024 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.685856 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.686024 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.685865 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2qg\" (UniqueName: \"kubernetes.io/projected/83b3ebbb-20a1-442c-8ed7-7c930929d773-kube-api-access-9d2qg\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.686024 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.685875 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b3ebbb-20a1-442c-8ed7-7c930929d773-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:06:36.995949 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.995914 4868 generic.go:356] "Generic (PLEG): container finished" podID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerID="4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4" exitCode=0 Nov 02 04:06:36.996112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.995961 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" Nov 02 04:06:36.996112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.995998 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" event={"ID":"83b3ebbb-20a1-442c-8ed7-7c930929d773","Type":"ContainerDied","Data":"4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4"} Nov 02 04:06:36.996112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.996075 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2" event={"ID":"83b3ebbb-20a1-442c-8ed7-7c930929d773","Type":"ContainerDied","Data":"7ef05c9a4777c5ecc46f8ac58fd321bbb6bde1c530812c272320c3c64c53fabf"} Nov 02 04:06:36.996112 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:36.996098 4868 scope.go:122] "RemoveContainer" containerID="4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4" Nov 02 04:06:37.006480 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.006463 4868 scope.go:122] "RemoveContainer" containerID="616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238" Nov 02 04:06:37.017269 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.017253 4868 scope.go:122] "RemoveContainer" containerID="4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4" Nov 02 04:06:37.017581 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:06:37.017556 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4\": container with ID starting with 4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4 not found: ID does not exist" containerID="4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4" Nov 02 04:06:37.017662 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.017588 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4"} err="failed to get container status \"4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4\": rpc error: code = NotFound desc = could not find container \"4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4\": container with ID starting with 4b0c6fe85105b8eb55178145bee98522aec9c3fc2f87cc630092ef9d43cca5c4 not found: ID does not exist" Nov 02 04:06:37.017662 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.017607 4868 scope.go:122] "RemoveContainer" containerID="616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238" Nov 02 04:06:37.017867 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:06:37.017850 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238\": container with ID starting with 616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238 not found: ID does not exist" containerID="616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238" Nov 02 04:06:37.017911 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.017873 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238"} err="failed to get container status \"616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238\": rpc error: code = NotFound desc = could not find container \"616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238\": container with ID starting with 616f980f97145f245fff1b081433fed9379b47c09aeb4e56f20881ef22568238 not found: ID does not exist" Nov 02 04:06:37.159189 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.159165 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2"] Nov 02 04:06:37.208456 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:37.208435 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5ac93f-openshift-default-77d76bbd85-47pn2"] Nov 02 04:06:38.116045 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:38.116011 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" path="/var/lib/kubelet/pods/83b3ebbb-20a1-442c-8ed7-7c930929d773/volumes" Nov 02 04:06:40.108887 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:06:40.108855 4868 scope.go:122] "RemoveContainer" containerID="460a7b60b5f8dcb41780450c453f6fb37eaa0c31d84fabde6aebcef03165034d" Nov 02 04:07:03.445672 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:07:03.445629 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:07:24.258611 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:24.258567 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md"] Nov 02 04:07:29.259861 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.259814 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" podUID="6be50840-13ca-4a52-a9c1-f159af4e9312" containerName="payload-processing" containerID="cri-o://c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898" gracePeriod=25 Nov 02 04:07:29.502597 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.502573 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:07:29.649412 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.649381 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f\" (UniqueName: \"kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f\") pod \"6be50840-13ca-4a52-a9c1-f159af4e9312\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " Nov 02 04:07:29.649565 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.649479 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume\") pod \"6be50840-13ca-4a52-a9c1-f159af4e9312\" (UID: \"6be50840-13ca-4a52-a9c1-f159af4e9312\") " Nov 02 04:07:29.649966 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.649935 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume" pod "6be50840-13ca-4a52-a9c1-f159af4e9312" (UID: "6be50840-13ca-4a52-a9c1-f159af4e9312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:07:29.651606 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.651582 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f" pod "6be50840-13ca-4a52-a9c1-f159af4e9312" (UID: "6be50840-13ca-4a52-a9c1-f159af4e9312"). InnerVolumeSpecName "kube-api-access-4wd4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:07:29.750843 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.750811 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be50840-13ca-4a52-a9c1-f159af4e9312-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:07:29.750843 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:29.750841 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wd4f\" (UniqueName: \"kubernetes.io/projected/6be50840-13ca-4a52-a9c1-f159af4e9312-kube-api-access-4wd4f\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:07:30.300675 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.300644 4868 generic.go:356] "Generic (PLEG): container finished" podID="6be50840-13ca-4a52-a9c1-f159af4e9312" containerID="c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898" exitCode=0 Nov 02 04:07:30.301204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.300700 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" Nov 02 04:07:30.301204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.300731 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" event={"ID":"6be50840-13ca-4a52-a9c1-f159af4e9312","Type":"ContainerDied","Data":"c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898"} Nov 02 04:07:30.301204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.300783 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md" event={"ID":"6be50840-13ca-4a52-a9c1-f159af4e9312","Type":"ContainerDied","Data":"7c9ae346dacbec23f20df450be35156d7dc35aad369e7b1284cc48101df60155"} Nov 02 04:07:30.301204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.300806 4868 scope.go:122] "RemoveContainer" containerID="c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898" Nov 02 04:07:30.309763 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.309745 4868 scope.go:122] "RemoveContainer" containerID="c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898" Nov 02 04:07:30.310031 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:07:30.310013 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898\": container with ID starting with c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898 not found: ID does not exist" containerID="c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898" Nov 02 04:07:30.310077 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.310039 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898"} err="failed to get container status \"c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898\": rpc error: code = NotFound desc = could not find container \"c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898\": container with ID starting with c9df4d355ee92effca5a4686f8ab5442032c5c51e4ee239c9658008fd5c68898 not found: ID does not exist" Nov 02 04:07:30.463068 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.463006 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md"] Nov 02 04:07:30.467649 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:30.467627 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3f24d0-79ff68994f-sk2md"] Nov 02 04:07:32.113655 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:32.113618 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be50840-13ca-4a52-a9c1-f159af4e9312" path="/var/lib/kubelet/pods/6be50840-13ca-4a52-a9c1-f159af4e9312/volumes" Nov 02 04:07:33.446524 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:07:33.446492 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:07:38.737028 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:38.736997 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 04:07:38.749154 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:38.749128 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-k6f8b_971a0263-ee9a-4280-aa61-d385483ca8f0/console-operator/2.log" Nov 02 04:07:40.142877 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:40.142847 4868 scope.go:122] "RemoveContainer" containerID="94078eb0ce79fdc734d242c42ad6d5d08883f5ba03ba2c9d151d328bb73d2c71" Nov 02 04:07:40.156435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:40.156413 4868 scope.go:122] "RemoveContainer" containerID="ae8ce07d730d7ae22bc6bce5b947ac849ec77de92d6b4c9edd5cfe6f802ac66f" Nov 02 04:07:59.283952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:59.283875 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9"] Nov 02 04:07:59.284502 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:07:59.284100 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" containerID="cri-o://d7bf238f0f2ce9f9064db3f520c6ef8b9d7c3fb7bec34f196c779cdeb7755f81" gracePeriod=30 Nov 02 04:08:03.444419 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:08:03.444378 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:08:04.489374 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.489322 4868 generic.go:356] "Generic (PLEG): container finished" podID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerID="d7bf238f0f2ce9f9064db3f520c6ef8b9d7c3fb7bec34f196c779cdeb7755f81" exitCode=0 Nov 02 04:08:04.489813 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.489385 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" event={"ID":"e2693bf6-5c99-45ed-bed7-605704705d8f","Type":"ContainerDied","Data":"d7bf238f0f2ce9f9064db3f520c6ef8b9d7c3fb7bec34f196c779cdeb7755f81"} Nov 02 04:08:04.489813 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.489465 4868 scope.go:122] "RemoveContainer" containerID="6dbc89d926e8146143175e592a9128962fa902c7c499db1dbd72857b0843bf63" Nov 02 04:08:04.545952 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.545931 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:08:04.656327 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656302 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656337 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656388 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656428 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656454 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656476 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656509 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656535 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.656588 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656577 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct\") pod \"e2693bf6-5c99-45ed-bed7-605704705d8f\" (UID: \"e2693bf6-5c99-45ed-bed7-605704705d8f\") " Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656742 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656757 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656823 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656879 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656945 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656966 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.656982 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.657046 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.657018 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:08:04.658711 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.658668 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:08:04.658836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.658775 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "kube-api-access-lpcct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:08:04.658836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.658806 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:08:04.658836 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.658829 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy" pod "e2693bf6-5c99-45ed-bed7-605704705d8f" (UID: "e2693bf6-5c99-45ed-bed7-605704705d8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:08:04.757899 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.757869 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.757899 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.757894 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2693bf6-5c99-45ed-bed7-605704705d8f-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.757899 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.757904 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.758104 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.757913 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.758104 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.757922 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:04.758104 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:04.757931 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcct\" (UniqueName: \"kubernetes.io/projected/e2693bf6-5c99-45ed-bed7-605704705d8f-kube-api-access-lpcct\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:08:05.496114 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:05.496082 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" Nov 02 04:08:05.496589 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:05.496082 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9" event={"ID":"e2693bf6-5c99-45ed-bed7-605704705d8f","Type":"ContainerDied","Data":"2cfc7e5391fabe41a8e7f29218ff116096c71e2f141a6d23ad805f36207c984f"} Nov 02 04:08:05.496589 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:05.496214 4868 scope.go:122] "RemoveContainer" containerID="d7bf238f0f2ce9f9064db3f520c6ef8b9d7c3fb7bec34f196c779cdeb7755f81" Nov 02 04:08:05.705106 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:05.705075 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9"] Nov 02 04:08:05.754403 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:05.754333 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3f24d0-openshift-default-6c878989f7-n2st9"] Nov 02 04:08:06.114197 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:06.114160 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" path="/var/lib/kubelet/pods/e2693bf6-5c99-45ed-bed7-605704705d8f/volumes" Nov 02 04:08:33.444827 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:08:33.444789 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:08:40.184569 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:40.184538 4868 scope.go:122] "RemoveContainer" containerID="1cbe7153c24a8a8751d6bfb044af630456c55d2fb6e84091e79c980d05aefd93" Nov 02 04:08:40.193127 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:40.193106 4868 scope.go:122] "RemoveContainer" containerID="10f87c58ed07108ac452f20255105f14822a25ce6eca1ccced0b6f706b2e1bdb" Nov 02 04:08:40.200992 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:40.200974 4868 scope.go:122] "RemoveContainer" containerID="225ad3964235473221db6a4bc9f1f8281553762c353ae3c39d130378204951c8" Nov 02 04:08:40.208628 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:08:40.208609 4868 scope.go:122] "RemoveContainer" containerID="411e1c50d6a7f6a0ca17cc72deb0ed88c9a95a0608d94d56744d880cb09c30d6" Nov 02 04:09:03.444935 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:03.444897 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:09:17.773919 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:17.773884 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv"] Nov 02 04:09:22.774931 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:22.774834 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerName="payload-processing" containerID="cri-o://25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d" gracePeriod=25 Nov 02 04:09:27.669562 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:27.669522 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:09:29.234446 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:29.234408 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r"] Nov 02 04:09:29.234874 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:29.234625 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" containerID="cri-o://ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330" gracePeriod=30 Nov 02 04:09:33.444294 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:33.444257 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:09:34.496519 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.496463 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 04:09:34.532730 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.532704 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.532825 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.532754 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.532825 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.532787 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.532938 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.532898 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.532996 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.532945 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.533053 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533014 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.533113 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533082 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:34.533178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533099 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:34.533178 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533131 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.533299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533205 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.533299 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533236 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket\") pod \"a01c3224-7761-4203-858f-c7934ed114fb\" (UID: \"a01c3224-7761-4203-858f-c7934ed114fb\") " Nov 02 04:09:34.533425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533340 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:09:34.533602 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533581 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:34.533688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533645 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.533688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533667 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.533688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533669 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:34.533688 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.533682 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01c3224-7761-4203-858f-c7934ed114fb-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.535062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.535029 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:09:34.535212 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.535186 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:34.535536 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.535521 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:34.535606 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.535560 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94" pod "a01c3224-7761-4203-858f-c7934ed114fb" (UID: "a01c3224-7761-4203-858f-c7934ed114fb"). InnerVolumeSpecName "kube-api-access-8gp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:34.634888 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.634865 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.634888 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.634885 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.635021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.634894 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01c3224-7761-4203-858f-c7934ed114fb-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.635021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.634904 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gp94\" (UniqueName: \"kubernetes.io/projected/a01c3224-7761-4203-858f-c7934ed114fb-kube-api-access-8gp94\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.635021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.634914 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.635021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.634922 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01c3224-7761-4203-858f-c7934ed114fb-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:34.658749 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.658726 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7"] Nov 02 04:09:34.710479 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.710449 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s"] Nov 02 04:09:34.715595 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.715575 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns"] Nov 02 04:09:34.988405 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.988326 4868 generic.go:356] "Generic (PLEG): container finished" podID="a01c3224-7761-4203-858f-c7934ed114fb" containerID="ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330" exitCode=0 Nov 02 04:09:34.988405 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.988398 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" Nov 02 04:09:34.988681 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.988408 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" event={"ID":"a01c3224-7761-4203-858f-c7934ed114fb","Type":"ContainerDied","Data":"ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330"} Nov 02 04:09:34.988681 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.988460 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r" event={"ID":"a01c3224-7761-4203-858f-c7934ed114fb","Type":"ContainerDied","Data":"a6dcc89dc0ec0efa80eb90f0eaff7a975ea8291053eda6e1a41fbcd26d44405f"} Nov 02 04:09:34.988681 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.988486 4868 scope.go:122] "RemoveContainer" containerID="ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330" Nov 02 04:09:34.998367 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:34.998333 4868 scope.go:122] "RemoveContainer" containerID="663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689" Nov 02 04:09:35.007896 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:35.007876 4868 scope.go:122] "RemoveContainer" containerID="ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330" Nov 02 04:09:35.008189 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:35.008170 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330\": container with ID starting with ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330 not found: ID does not exist" containerID="ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330" Nov 02 04:09:35.008246 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:35.008198 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330"} err="failed to get container status \"ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330\": rpc error: code = NotFound desc = could not find container \"ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330\": container with ID starting with ba1d22a48d492b7802cbe4c116e96b90b5a9f436ad6885077de785cf7f03f330 not found: ID does not exist" Nov 02 04:09:35.008246 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:35.008218 4868 scope.go:122] "RemoveContainer" containerID="663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689" Nov 02 04:09:35.008493 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:35.008475 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689\": container with ID starting with 663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689 not found: ID does not exist" containerID="663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689" Nov 02 04:09:35.008551 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:35.008498 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689"} err="failed to get container status \"663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689\": rpc error: code = NotFound desc = could not find container \"663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689\": container with ID starting with 663855aed21e2e4f293babc9efd5a02ff556485a8ce00838181604eea2609689 not found: ID does not exist" Nov 02 04:09:35.266835 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:35.266754 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r"] Nov 02 04:09:35.274758 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:35.274728 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3d14ee-openshift-default-56d864fcd7-4xm4r"] Nov 02 04:09:36.112840 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:36.112801 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01c3224-7761-4203-858f-c7934ed114fb" path="/var/lib/kubelet/pods/a01c3224-7761-4203-858f-c7934ed114fb/volumes" Nov 02 04:09:37.669523 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:37.669485 4868 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 04:09:40.243280 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:40.243247 4868 scope.go:122] "RemoveContainer" containerID="ee145596beb0dfdc9ad6861fd563b96037c4490c71c762cc9d4c4d9eb7884e3c" Nov 02 04:09:40.252336 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:40.252318 4868 scope.go:122] "RemoveContainer" containerID="bb9016d71fcd37cd22626b9ac4c3ae53bab1544082dc172659a8a0c41137d2be" Nov 02 04:09:40.260461 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:40.260444 4868 scope.go:122] "RemoveContainer" containerID="260e50f66388af44c551e916f6041d15b840a3cd40549232ae97eae5a1c9c036" Nov 02 04:09:42.415366 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:42.415330 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz"] Nov 02 04:09:43.024189 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.024165 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 04:09:43.039857 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.039788 4868 generic.go:356] "Generic (PLEG): container finished" podID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerID="25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d" exitCode=0 Nov 02 04:09:43.039857 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.039823 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" event={"ID":"cd6422fe-c398-4865-8bf2-b83df32106d0","Type":"ContainerDied","Data":"25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d"} Nov 02 04:09:43.040028 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.039862 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" Nov 02 04:09:43.040028 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.039923 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv" event={"ID":"cd6422fe-c398-4865-8bf2-b83df32106d0","Type":"ContainerDied","Data":"14f4b4d90baecb68a835697cf3d9c630b7ad3aba998249f241053b8f8ef0a0b3"} Nov 02 04:09:43.040028 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.039939 4868 scope.go:122] "RemoveContainer" containerID="25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d" Nov 02 04:09:43.040194 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.040142 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" containerID="cri-o://6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566" gracePeriod=30 Nov 02 04:09:43.049322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.049307 4868 scope.go:122] "RemoveContainer" containerID="25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d" Nov 02 04:09:43.049607 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:43.049591 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d\": container with ID starting with 25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d not found: ID does not exist" containerID="25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d" Nov 02 04:09:43.049662 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.049614 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d"} err="failed to get container status \"25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d\": rpc error: code = NotFound desc = could not find container \"25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d\": container with ID starting with 25f61b5050a0ed792d55e6278a967690d2b9a3ff8dee87a0929ad23002be459d not found: ID does not exist" Nov 02 04:09:43.103394 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.103369 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx\" (UniqueName: \"kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx\") pod \"cd6422fe-c398-4865-8bf2-b83df32106d0\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " Nov 02 04:09:43.103502 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.103488 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume\") pod \"cd6422fe-c398-4865-8bf2-b83df32106d0\" (UID: \"cd6422fe-c398-4865-8bf2-b83df32106d0\") " Nov 02 04:09:43.103833 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.103817 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume" pod "cd6422fe-c398-4865-8bf2-b83df32106d0" (UID: "cd6422fe-c398-4865-8bf2-b83df32106d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:09:43.105240 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.105213 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx" pod "cd6422fe-c398-4865-8bf2-b83df32106d0" (UID: "cd6422fe-c398-4865-8bf2-b83df32106d0"). InnerVolumeSpecName "kube-api-access-4qvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:43.204220 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.204190 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvnx\" (UniqueName: \"kubernetes.io/projected/cd6422fe-c398-4865-8bf2-b83df32106d0-kube-api-access-4qvnx\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:43.204220 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.204216 4868 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6422fe-c398-4865-8bf2-b83df32106d0-plugins-config-volume\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:43.463707 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.463677 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv"] Nov 02 04:09:43.467154 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:43.467114 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3d14ee-79cf5597df-r67vv"] Nov 02 04:09:44.113455 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:44.113415 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" path="/var/lib/kubelet/pods/cd6422fe-c398-4865-8bf2-b83df32106d0/volumes" Nov 02 04:09:48.280636 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.280613 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:09:48.348427 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348373 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348427 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348420 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348457 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348480 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348510 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348582 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348550 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348805 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348708 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348805 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348770 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348917 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348797 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:48.348917 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348832 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs\") pod \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\" (UID: \"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a\") " Nov 02 04:09:48.348917 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348884 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:48.348917 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.348893 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:09:48.349148 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.349052 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:48.349216 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.349169 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:48.349322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.349212 4868 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.349322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.349233 4868 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-workload-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.349322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.349248 4868 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istiod-ca-cert\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.349322 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.349263 4868 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-data\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.350605 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.350580 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:48.350725 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.350636 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:48.351014 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.350998 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "kube-api-access-mdfzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:48.351129 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.351110 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo" pod "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" (UID: "b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 04:09:48.450077 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.450054 4868 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-credential-socket\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.450077 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.450078 4868 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-podinfo\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.450214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.450088 4868 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-token\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.450214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.450097 4868 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-istio-envoy\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:48.450214 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:48.450107 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfzn\" (UniqueName: \"kubernetes.io/projected/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a-kube-api-access-mdfzn\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:49.076280 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.076249 4868 generic.go:356] "Generic (PLEG): container finished" podID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerID="6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566" exitCode=0 Nov 02 04:09:49.076489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.076313 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" Nov 02 04:09:49.076489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.076329 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" event={"ID":"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a","Type":"ContainerDied","Data":"6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566"} Nov 02 04:09:49.076489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.076392 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz" event={"ID":"b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a","Type":"ContainerDied","Data":"b8a4ef92188f0acd6b62ac9baf06b9fe67e474d57e7ec28d41476ed4f5a06aa0"} Nov 02 04:09:49.076489 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.076414 4868 scope.go:122] "RemoveContainer" containerID="6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566" Nov 02 04:09:49.086425 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.086408 4868 scope.go:122] "RemoveContainer" containerID="6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566" Nov 02 04:09:49.086706 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:49.086680 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566\": container with ID starting with 6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566 not found: ID does not exist" containerID="6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566" Nov 02 04:09:49.086762 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.086717 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566"} err="failed to get container status \"6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566\": rpc error: code = NotFound desc = could not find container \"6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566\": container with ID starting with 6771a3ea19a0f6e71e7c076dd5448154c8454cbd82109df5b755f23d9aac4566 not found: ID does not exist" Nov 02 04:09:49.292622 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.292588 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz"] Nov 02 04:09:49.341927 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.341859 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mtpnz"] Nov 02 04:09:49.677532 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.677496 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" containerID="cri-o://a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e" gracePeriod=45 Nov 02 04:09:49.734556 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.734507 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" containerID="cri-o://62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6" gracePeriod=45 Nov 02 04:09:49.978761 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.978735 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 04:09:49.998250 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:49.998226 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 04:09:50.009216 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.009186 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" containerID="cri-o://00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a" gracePeriod=45 Nov 02 04:09:50.063980 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.063949 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm\") pod \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " Nov 02 04:09:50.064117 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064010 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97\" (UniqueName: \"kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97\") pod \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " Nov 02 04:09:50.064117 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064043 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache\") pod \"eeef3ea2-666b-406e-90f2-a059512f1a90\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " Nov 02 04:09:50.064117 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064078 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs\") pod \"eeef3ea2-666b-406e-90f2-a059512f1a90\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " Nov 02 04:09:50.064117 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064114 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs\") pod \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " Nov 02 04:09:50.064383 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064138 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm\") pod \"eeef3ea2-666b-406e-90f2-a059512f1a90\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " Nov 02 04:09:50.064383 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064215 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home\") pod \"eeef3ea2-666b-406e-90f2-a059512f1a90\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " Nov 02 04:09:50.064383 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064248 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir\") pod \"eeef3ea2-666b-406e-90f2-a059512f1a90\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " Nov 02 04:09:50.064383 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064332 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache" pod "eeef3ea2-666b-406e-90f2-a059512f1a90" (UID: "eeef3ea2-666b-406e-90f2-a059512f1a90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.064610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064442 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home\") pod \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " Nov 02 04:09:50.064610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064524 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache\") pod \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " Nov 02 04:09:50.064610 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064569 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir" pod "eeef3ea2-666b-406e-90f2-a059512f1a90" (UID: "eeef3ea2-666b-406e-90f2-a059512f1a90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.064787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064665 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir\") pod \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\" (UID: \"f75d7404-a123-4115-bd4d-3a1ff3bf77e2\") " Nov 02 04:09:50.064787 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064759 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home" pod "eeef3ea2-666b-406e-90f2-a059512f1a90" (UID: "eeef3ea2-666b-406e-90f2-a059512f1a90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.064904 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064761 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home" pod "f75d7404-a123-4115-bd4d-3a1ff3bf77e2" (UID: "f75d7404-a123-4115-bd4d-3a1ff3bf77e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.064904 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064761 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8\" (UniqueName: \"kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8\") pod \"eeef3ea2-666b-406e-90f2-a059512f1a90\" (UID: \"eeef3ea2-666b-406e-90f2-a059512f1a90\") " Nov 02 04:09:50.065034 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.064957 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache" pod "f75d7404-a123-4115-bd4d-3a1ff3bf77e2" (UID: "f75d7404-a123-4115-bd4d-3a1ff3bf77e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.065150 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065129 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir" pod "f75d7404-a123-4115-bd4d-3a1ff3bf77e2" (UID: "f75d7404-a123-4115-bd4d-3a1ff3bf77e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.065297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065279 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.065424 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065301 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.065424 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065314 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.065424 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065326 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.065424 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065340 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.065424 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.065369 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.066999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.066966 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97" pod "f75d7404-a123-4115-bd4d-3a1ff3bf77e2" (UID: "f75d7404-a123-4115-bd4d-3a1ff3bf77e2"). InnerVolumeSpecName "kube-api-access-dsd97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:50.066999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.066982 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm" pod "f75d7404-a123-4115-bd4d-3a1ff3bf77e2" (UID: "f75d7404-a123-4115-bd4d-3a1ff3bf77e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.067200 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.067080 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8" pod "eeef3ea2-666b-406e-90f2-a059512f1a90" (UID: "eeef3ea2-666b-406e-90f2-a059512f1a90"). InnerVolumeSpecName "kube-api-access-cm9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:50.067200 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.067091 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs" pod "f75d7404-a123-4115-bd4d-3a1ff3bf77e2" (UID: "f75d7404-a123-4115-bd4d-3a1ff3bf77e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:09:50.067311 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.067240 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs" pod "eeef3ea2-666b-406e-90f2-a059512f1a90" (UID: "eeef3ea2-666b-406e-90f2-a059512f1a90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:09:50.067439 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.067411 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm" pod "eeef3ea2-666b-406e-90f2-a059512f1a90" (UID: "eeef3ea2-666b-406e-90f2-a059512f1a90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.083020 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.082990 4868 generic.go:356] "Generic (PLEG): container finished" podID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerID="62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6" exitCode=0 Nov 02 04:09:50.083140 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.083034 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" Nov 02 04:09:50.083140 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.083065 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" event={"ID":"eeef3ea2-666b-406e-90f2-a059512f1a90","Type":"ContainerDied","Data":"62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6"} Nov 02 04:09:50.083140 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.083106 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns" event={"ID":"eeef3ea2-666b-406e-90f2-a059512f1a90","Type":"ContainerDied","Data":"3e0c7fa500151cba37d5e0297851d61109bb9778f9815e1d58fad0f96e659d23"} Nov 02 04:09:50.083140 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.083125 4868 scope.go:122] "RemoveContainer" containerID="62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6" Nov 02 04:09:50.084495 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.084475 4868 generic.go:356] "Generic (PLEG): container finished" podID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerID="a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e" exitCode=0 Nov 02 04:09:50.084590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.084503 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" Nov 02 04:09:50.084590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.084541 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" event={"ID":"f75d7404-a123-4115-bd4d-3a1ff3bf77e2","Type":"ContainerDied","Data":"a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e"} Nov 02 04:09:50.084590 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.084572 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7" event={"ID":"f75d7404-a123-4115-bd4d-3a1ff3bf77e2","Type":"ContainerDied","Data":"af2bcf729259036bbff11f17614bf9dc8166a93227822e274ed686bb8efcdb6e"} Nov 02 04:09:50.114492 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.114467 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" path="/var/lib/kubelet/pods/b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a/volumes" Nov 02 04:09:50.122091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.122067 4868 scope.go:122] "RemoveContainer" containerID="62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6" Nov 02 04:09:50.122432 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:50.122405 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6\": container with ID starting with 62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6 not found: ID does not exist" containerID="62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6" Nov 02 04:09:50.122549 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.122438 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6"} err="failed to get container status \"62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6\": rpc error: code = NotFound desc = could not find container \"62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6\": container with ID starting with 62bbd6d85e004caea97faef7eb5379993ef453f107e05ee483daadae7e133fb6 not found: ID does not exist" Nov 02 04:09:50.122549 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.122455 4868 scope.go:122] "RemoveContainer" containerID="a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e" Nov 02 04:09:50.130762 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.130746 4868 scope.go:122] "RemoveContainer" containerID="a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e" Nov 02 04:09:50.131056 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:50.131032 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e\": container with ID starting with a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e not found: ID does not exist" containerID="a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e" Nov 02 04:09:50.131107 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.131067 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e"} err="failed to get container status \"a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e\": rpc error: code = NotFound desc = could not find container \"a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e\": container with ID starting with a650048e0f7ce13cda2a1a9a84b164d0c06b6b718fa2e4b1c836b83d9713bf5e not found: ID does not exist" Nov 02 04:09:50.166183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.166154 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.166183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.166178 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsd97\" (UniqueName: \"kubernetes.io/projected/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-kube-api-access-dsd97\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.166183 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.166189 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeef3ea2-666b-406e-90f2-a059512f1a90-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.166658 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.166198 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f75d7404-a123-4115-bd4d-3a1ff3bf77e2-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.166658 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.166207 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeef3ea2-666b-406e-90f2-a059512f1a90-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.166658 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.166214 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9k8\" (UniqueName: \"kubernetes.io/projected/eeef3ea2-666b-406e-90f2-a059512f1a90-kube-api-access-cm9k8\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.268691 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.268663 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 04:09:50.295146 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.295113 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns"] Nov 02 04:09:50.300690 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.300662 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-858236/simulator-w3-858236-kserve-755f99b74-t7jns"] Nov 02 04:09:50.367827 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.367801 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache\") pod \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " Nov 02 04:09:50.367970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.367833 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir\") pod \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " Nov 02 04:09:50.367970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.367902 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs\") pod \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " Nov 02 04:09:50.367970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.367929 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm\") pod \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " Nov 02 04:09:50.367970 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.367970 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home\") pod \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " Nov 02 04:09:50.368197 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.367999 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk\" (UniqueName: \"kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk\") pod \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\" (UID: \"367ac7f7-72db-466f-a2b9-8a45e54f32d8\") " Nov 02 04:09:50.368197 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.368060 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache" pod "367ac7f7-72db-466f-a2b9-8a45e54f32d8" (UID: "367ac7f7-72db-466f-a2b9-8a45e54f32d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.368197 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.368176 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir" pod "367ac7f7-72db-466f-a2b9-8a45e54f32d8" (UID: "367ac7f7-72db-466f-a2b9-8a45e54f32d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.368406 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.368247 4868 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-model-cache\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.368406 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.368259 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home" pod "367ac7f7-72db-466f-a2b9-8a45e54f32d8" (UID: "367ac7f7-72db-466f-a2b9-8a45e54f32d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.370100 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.370077 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs" pod "367ac7f7-72db-466f-a2b9-8a45e54f32d8" (UID: "367ac7f7-72db-466f-a2b9-8a45e54f32d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:09:50.370218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.370082 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm" pod "367ac7f7-72db-466f-a2b9-8a45e54f32d8" (UID: "367ac7f7-72db-466f-a2b9-8a45e54f32d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 04:09:50.370218 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.370110 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk" pod "367ac7f7-72db-466f-a2b9-8a45e54f32d8" (UID: "367ac7f7-72db-466f-a2b9-8a45e54f32d8"). InnerVolumeSpecName "kube-api-access-nqlwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:09:50.408760 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.408735 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7"] Nov 02 04:09:50.412703 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.412683 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-858236/distinct-2-w3-858236-kserve-5c4b874cd5-fmgf7"] Nov 02 04:09:50.469297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.469250 4868 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tls-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.469297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.469269 4868 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-dshm\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.469297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.469278 4868 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-home\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.469297 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.469288 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqlwk\" (UniqueName: \"kubernetes.io/projected/367ac7f7-72db-466f-a2b9-8a45e54f32d8-kube-api-access-nqlwk\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:50.469537 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:50.469316 4868 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/367ac7f7-72db-466f-a2b9-8a45e54f32d8-tmp-dir\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:09:51.092771 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.092726 4868 generic.go:356] "Generic (PLEG): container finished" podID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerID="00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a" exitCode=0 Nov 02 04:09:51.092771 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.092776 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" Nov 02 04:09:51.093157 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.092810 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" event={"ID":"367ac7f7-72db-466f-a2b9-8a45e54f32d8","Type":"ContainerDied","Data":"00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a"} Nov 02 04:09:51.093157 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.092855 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s" event={"ID":"367ac7f7-72db-466f-a2b9-8a45e54f32d8","Type":"ContainerDied","Data":"df04693ed4ae02470f4ed631a70f498cad0d8a277c8044d417565dfd2d0636be"} Nov 02 04:09:51.093157 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.092880 4868 scope.go:122] "RemoveContainer" containerID="00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a" Nov 02 04:09:51.103945 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.103924 4868 scope.go:122] "RemoveContainer" containerID="00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a" Nov 02 04:09:51.104213 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:09:51.104194 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a\": container with ID starting with 00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a not found: ID does not exist" containerID="00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a" Nov 02 04:09:51.104278 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.104226 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a"} err="failed to get container status \"00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a\": rpc error: code = NotFound desc = could not find container \"00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a\": container with ID starting with 00503a95ea5418276fa59f4be4eb61e8adb8aef4fbe1ce558230d701b9ed377a not found: ID does not exist" Nov 02 04:09:51.255615 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.255585 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s"] Nov 02 04:09:51.304690 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:51.304664 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-858236/premium-w3-858236-kserve-54dd8c6599-ld76s"] Nov 02 04:09:52.112639 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:52.112607 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" path="/var/lib/kubelet/pods/367ac7f7-72db-466f-a2b9-8a45e54f32d8/volumes" Nov 02 04:09:52.113013 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:52.112998 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" path="/var/lib/kubelet/pods/eeef3ea2-666b-406e-90f2-a059512f1a90/volumes" Nov 02 04:09:52.113376 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:52.113361 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" path="/var/lib/kubelet/pods/f75d7404-a123-4115-bd4d-3a1ff3bf77e2/volumes" Nov 02 04:09:54.050811 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:54.050774 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj"] Nov 02 04:09:54.101021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:54.100993 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc"] Nov 02 04:09:54.102466 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:54.102445 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a8972b-dxkdj"] Nov 02 04:09:54.103666 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:54.103645 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21d5110-6c7tc"] Nov 02 04:09:54.113469 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:54.113443 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215bde34-daec-4af3-baed-4eb1bf4bd191" path="/var/lib/kubelet/pods/215bde34-daec-4af3-baed-4eb1bf4bd191/volumes" Nov 02 04:09:54.113857 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:09:54.113839 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c81922-d957-4a89-8b5d-bda90db91028" path="/var/lib/kubelet/pods/50c81922-d957-4a89-8b5d-bda90db91028/volumes" Nov 02 04:10:03.445079 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:10:03.445043 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:10:04.330987 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.330955 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c"] Nov 02 04:10:04.331204 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.331176 4868 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" podUID="b5811b16-e35e-450f-bf28-543513ce47da" containerName="maas-api" containerID="cri-o://1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b" gracePeriod=30 Nov 02 04:10:04.585122 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.585067 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 04:10:04.683091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.681902 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca\") pod \"b5811b16-e35e-450f-bf28-543513ce47da\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " Nov 02 04:10:04.683091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.682027 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs\") pod \"b5811b16-e35e-450f-bf28-543513ce47da\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " Nov 02 04:10:04.683091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.682095 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx\" (UniqueName: \"kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx\") pod \"b5811b16-e35e-450f-bf28-543513ce47da\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " Nov 02 04:10:04.683091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.682144 4868 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\") pod \"b5811b16-e35e-450f-bf28-543513ce47da\" (UID: \"b5811b16-e35e-450f-bf28-543513ce47da\") " Nov 02 04:10:04.683091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.682156 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca" pod "b5811b16-e35e-450f-bf28-543513ce47da" (UID: "b5811b16-e35e-450f-bf28-543513ce47da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 04:10:04.683091 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.682609 4868 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5811b16-e35e-450f-bf28-543513ce47da-openshift-service-ca\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:10:04.686487 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.686055 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls" pod "b5811b16-e35e-450f-bf28-543513ce47da" (UID: "b5811b16-e35e-450f-bf28-543513ce47da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:10:04.686632 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.686544 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs" pod "b5811b16-e35e-450f-bf28-543513ce47da" (UID: "b5811b16-e35e-450f-bf28-543513ce47da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 04:10:04.689435 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.689404 4868 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx" pod "b5811b16-e35e-450f-bf28-543513ce47da" (UID: "b5811b16-e35e-450f-bf28-543513ce47da"). InnerVolumeSpecName "kube-api-access-xqgjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 04:10:04.783772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.783738 4868 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-maas-api-tls\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:10:04.783772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.783765 4868 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5811b16-e35e-450f-bf28-543513ce47da-metrics-certs\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:10:04.783772 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:04.783777 4868 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqgjx\" (UniqueName: \"kubernetes.io/projected/b5811b16-e35e-450f-bf28-543513ce47da-kube-api-access-xqgjx\") on node \"ip-10-0-137-74.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 04:10:05.164745 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.164710 4868 generic.go:356] "Generic (PLEG): container finished" podID="b5811b16-e35e-450f-bf28-543513ce47da" containerID="1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b" exitCode=0 Nov 02 04:10:05.164927 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.164759 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" event={"ID":"b5811b16-e35e-450f-bf28-543513ce47da","Type":"ContainerDied","Data":"1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b"} Nov 02 04:10:05.164927 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.164769 4868 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" Nov 02 04:10:05.164927 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.164795 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c" event={"ID":"b5811b16-e35e-450f-bf28-543513ce47da","Type":"ContainerDied","Data":"595cc1646da51a3f97c85eb5c7e2d70bd6d5821d1482c7cceb63b0cf12d05a22"} Nov 02 04:10:05.164927 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.164816 4868 scope.go:122] "RemoveContainer" containerID="1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b" Nov 02 04:10:05.174347 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.174333 4868 scope.go:122] "RemoveContainer" containerID="1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b" Nov 02 04:10:05.174618 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:10:05.174598 4868 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b\": container with ID starting with 1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b not found: ID does not exist" containerID="1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b" Nov 02 04:10:05.174698 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.174624 4868 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b"} err="failed to get container status \"1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b\": rpc error: code = NotFound desc = could not find container \"1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b\": container with ID starting with 1ef674a8e1b03b3f59b86803d83c7fbc93bb8a7892346516144c13cce9cfd79b not found: ID does not exist" Nov 02 04:10:05.327102 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.327068 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c"] Nov 02 04:10:05.376629 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:05.376596 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-858236-5bd4c9bd5-rcc7c"] Nov 02 04:10:06.113595 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:06.113560 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5811b16-e35e-450f-bf28-543513ce47da" path="/var/lib/kubelet/pods/b5811b16-e35e-450f-bf28-543513ce47da/volumes" Nov 02 04:10:11.046411 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:11.046374 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f"] Nov 02 04:10:11.087819 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:11.087790 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb"] Nov 02 04:10:11.095899 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:11.095871 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe48b66-qjm8f"] Nov 02 04:10:11.097146 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:11.097123 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9237d21c-wvswb"] Nov 02 04:10:12.113469 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:12.113437 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a5aaeb-0d21-4c1c-9a47-343b011b2bb8" path="/var/lib/kubelet/pods/20a5aaeb-0d21-4c1c-9a47-343b011b2bb8/volumes" Nov 02 04:10:12.113941 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:12.113911 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d013783-f450-41cd-ae4e-30d0b366233f" path="/var/lib/kubelet/pods/7d013783-f450-41cd-ae4e-30d0b366233f/volumes" Nov 02 04:10:15.315802 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:15.315773 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-84qbj_cce70044-e2d2-40aa-a55b-f1ba8f1cc976/kuadrant-console-plugin/0.log" Nov 02 04:10:18.501147 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:18.501112 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6spg5_e07d60c1-9802-4fa0-9a5b-85ec83152e37/manager/0.log" Nov 02 04:10:18.896018 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:18.895985 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dbf54fd5c-ttkx8_f5e7e1a3-15ef-485e-a84e-a257d62e8f14/manager/0.log" Nov 02 04:10:22.046015 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:22.045949 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs"] Nov 02 04:10:22.084335 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:22.084304 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949a867c-dmdfs"] Nov 02 04:10:22.094984 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:22.094960 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq"] Nov 02 04:10:22.096268 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:22.096247 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee7e627-9zttq"] Nov 02 04:10:22.114027 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:22.113998 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="524fa02f-4485-4aea-92c4-aa703da40e07" path="/var/lib/kubelet/pods/524fa02f-4485-4aea-92c4-aa703da40e07/volumes" Nov 02 04:10:22.114462 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:22.114446 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42cc639-7f8c-49ef-a88f-0d2d32e7c985" path="/var/lib/kubelet/pods/f42cc639-7f8c-49ef-a88f-0d2d32e7c985/volumes" Nov 02 04:10:31.113777 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:31.113744 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6spg5_e07d60c1-9802-4fa0-9a5b-85ec83152e37/manager/0.log" Nov 02 04:10:31.584696 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:31.584624 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dbf54fd5c-ttkx8_f5e7e1a3-15ef-485e-a84e-a257d62e8f14/manager/0.log" Nov 02 04:10:33.444756 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:10:33.444722 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:10:33.889905 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:33.889878 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6dbf92f1-j9ttj_172e64fa-d2c6-4dbc-b571-80fb94dc221c/revoke-keys/0.log" Nov 02 04:10:34.303258 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:34.303183 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-pgtf8_401ad747-9fed-4a52-a560-0bab16515219/postgres/0.log" Nov 02 04:10:36.542141 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:36.542110 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-84qbj_cce70044-e2d2-40aa-a55b-f1ba8f1cc976/kuadrant-console-plugin/0.log" Nov 02 04:10:38.992003 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:38.991977 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lqm9w_e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9/discovery/0.log" Nov 02 04:10:39.422150 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:39.422124 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66599bc4bb-z7fp2_8e6b08ff-10f3-4655-b67b-18931ee4162a/kube-auth-proxy/0.log" Nov 02 04:10:40.289693 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:40.289661 4868 scope.go:122] "RemoveContainer" containerID="fe4cebfe762285b897b9637340444b71bd20fa8b9a36bc1a57f0a462141738a0" Nov 02 04:10:40.298239 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:40.298215 4868 scope.go:122] "RemoveContainer" containerID="f8dfd2a78c514d884eefac4234f0b155bb833e7b762e41ff5185b2d215f00dcf" Nov 02 04:10:40.305928 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:40.305907 4868 scope.go:122] "RemoveContainer" containerID="6614cc357da8790d0a4bf18f210441309fd45022acc2bdd7c44be3f2645c10a5" Nov 02 04:10:40.313227 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:40.313212 4868 scope.go:122] "RemoveContainer" containerID="64bf6eee434cbaa7ab1f4d09fbb1f275ed84e793e856a08b2143c50c4b04ab26" Nov 02 04:10:40.537312 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:40.537285 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-648db889-4lftz_d25eaef0-b0ac-4326-925b-aaef13840661/router/0.log" Nov 02 04:10:56.054197 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:56.054114 4868 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj"] Nov 02 04:10:56.103293 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:56.103261 4868 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dbf92f1-j9ttj"] Nov 02 04:10:56.113523 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:10:56.113500 4868 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172e64fa-d2c6-4dbc-b571-80fb94dc221c" path="/var/lib/kubelet/pods/172e64fa-d2c6-4dbc-b571-80fb94dc221c/volumes" Nov 02 04:11:02.095983 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:02.095953 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xhmws_f2f5b5ce-fe6f-479e-ab0a-fc40bfaa1b5c/global-pull-secret-syncer/0.log" Nov 02 04:11:03.282085 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:03.282059 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ldnnl_d5c5e81a-fe80-4a82-bfbc-c094e1f8fde4/connection-checker/0.log" Nov 02 04:11:03.445823 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:11:03.445771 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:11:03.698231 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:03.698202 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-prxz2_76685eef-ba60-4361-9b35-c7291f159a8b/connection-checker/0.log" Nov 02 04:11:04.113899 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:04.113876 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-q69cv_e9bee798-853b-4ef3-b716-2e9bd203c71b/connection-checker/0.log" Nov 02 04:11:04.981124 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:04.981100 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v8xt2_3ecd4ba6-46cc-4f76-8984-492c35ad4f40/konnectivity-agent/0.log" Nov 02 04:11:05.403702 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:05.403669 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6spg5_e07d60c1-9802-4fa0-9a5b-85ec83152e37/manager/0.log" Nov 02 04:11:05.689131 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:05.689065 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dbf54fd5c-ttkx8_f5e7e1a3-15ef-485e-a84e-a257d62e8f14/manager/0.log" Nov 02 04:11:06.709999 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:06.709968 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-74.us-west-1.compute.internal_f403bae9d2c2c1c33d0488e61f8d37c6/haproxy/0.log" Nov 02 04:11:10.939863 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:10.939831 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6spg5_e07d60c1-9802-4fa0-9a5b-85ec83152e37/manager/0.log" Nov 02 04:11:11.259062 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.258966 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dbf54fd5c-ttkx8_f5e7e1a3-15ef-485e-a84e-a257d62e8f14/manager/0.log" Nov 02 04:11:11.628257 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628220 4868 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9"] Nov 02 04:11:11.628627 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628612 4868 cpu_manager.go:414] "containerMap: removing container" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerName="payload-processing" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628629 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerName="payload-processing" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628644 4868 cpu_manager.go:414] "containerMap: removing container" podUID="ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" containerName="istio-proxy" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628650 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" containerName="istio-proxy" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628664 4868 cpu_manager.go:414] "containerMap: removing container" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628669 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628680 4868 cpu_manager.go:414] "containerMap: removing container" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" Nov 02 04:11:11.628684 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628685 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628692 4868 cpu_manager.go:414] "containerMap: removing container" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628697 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628704 4868 cpu_manager.go:414] "containerMap: removing container" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628709 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628724 4868 cpu_manager.go:414] "containerMap: removing container" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628731 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628742 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628750 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628762 4868 cpu_manager.go:414] "containerMap: removing container" podUID="6be50840-13ca-4a52-a9c1-f159af4e9312" containerName="payload-processing" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628769 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be50840-13ca-4a52-a9c1-f159af4e9312" containerName="payload-processing" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628776 4868 cpu_manager.go:414] "containerMap: removing container" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628781 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628787 4868 cpu_manager.go:414] "containerMap: removing container" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628794 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628801 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b5811b16-e35e-450f-bf28-543513ce47da" containerName="maas-api" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628805 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5811b16-e35e-450f-bf28-543513ce47da" containerName="maas-api" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628815 4868 cpu_manager.go:414] "containerMap: removing container" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628819 4868 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628876 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b3ebbb-20a1-442c-8ed7-7c930929d773" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628886 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628892 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be50840-13ca-4a52-a9c1-f159af4e9312" containerName="payload-processing" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628898 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01c3224-7761-4203-858f-c7934ed114fb" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628903 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="367ac7f7-72db-466f-a2b9-8a45e54f32d8" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628908 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5811b16-e35e-450f-bf28-543513ce47da" containerName="maas-api" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628914 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628920 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ac9382-bad5-4bd0-9bb0-2dba8ba2206a" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628925 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6422fe-c398-4865-8bf2-b83df32106d0" containerName="payload-processing" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628931 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad66f25e-461a-4c36-bcaf-4dc59ad1f59d" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628938 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2693bf6-5c99-45ed-bed7-605704705d8f" containerName="istio-proxy" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628945 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="f75d7404-a123-4115-bd4d-3a1ff3bf77e2" containerName="main" Nov 02 04:11:11.628944 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.628950 4868 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeef3ea2-666b-406e-90f2-a059512f1a90" containerName="main" Nov 02 04:11:11.631729 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.631712 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.681332 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.681302 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-sys\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.681332 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.681327 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-proc\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.681507 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.681366 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-podres\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.681547 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.681524 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-lib-modules\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.681584 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.681557 4868 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxjq\" (UniqueName: \"kubernetes.io/projected/c7b37551-2bff-4657-a50b-4894c61dc3f4-kube-api-access-6xxjq\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.683501 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.683483 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmzv\"/\"openshift-service-ca.crt\"" Nov 02 04:11:11.693989 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.693963 4868 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmzv\"/\"kube-root-ca.crt\"" Nov 02 04:11:11.732978 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.732944 4868 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbmzv\"/\"default-dockercfg-c7lwr\"" Nov 02 04:11:11.734417 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.734396 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9"] Nov 02 04:11:11.782121 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782094 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-podres\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782274 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782168 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-lib-modules\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782274 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782199 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxjq\" (UniqueName: \"kubernetes.io/projected/c7b37551-2bff-4657-a50b-4894c61dc3f4-kube-api-access-6xxjq\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782274 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782232 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-podres\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782274 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782242 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-sys\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782288 4868 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-proc\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782301 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-sys\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782320 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-lib-modules\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.782533 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.782407 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7b37551-2bff-4657-a50b-4894c61dc3f4-proc\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.839374 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.839326 4868 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxjq\" (UniqueName: \"kubernetes.io/projected/c7b37551-2bff-4657-a50b-4894c61dc3f4-kube-api-access-6xxjq\") pod \"perf-node-gather-daemonset-jl9b9\" (UID: \"c7b37551-2bff-4657-a50b-4894c61dc3f4\") " pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:11.944606 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:11.944538 4868 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:12.235031 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:12.234927 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-84qbj_cce70044-e2d2-40aa-a55b-f1ba8f1cc976/kuadrant-console-plugin/0.log" Nov 02 04:11:12.319180 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:12.319144 4868 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9"] Nov 02 04:11:12.321907 ip-10-0-137-74 kubenswrapper[4868]: W1102 04:11:12.321880 4868 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7b37551_2bff_4657_a50b_4894c61dc3f4.slice/crio-8e20e2918f7b00dac0c846bff597f3e76af72f6ad2bc434ae3f6fcdf27f66c10 WatchSource:0}: Error finding container 8e20e2918f7b00dac0c846bff597f3e76af72f6ad2bc434ae3f6fcdf27f66c10: Status 404 returned error can't find the container with id 8e20e2918f7b00dac0c846bff597f3e76af72f6ad2bc434ae3f6fcdf27f66c10 Nov 02 04:11:12.485284 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:12.485222 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" event={"ID":"c7b37551-2bff-4657-a50b-4894c61dc3f4","Type":"ContainerStarted","Data":"7eb47a37a97a50631548ec8f0e7f834e27867450a16c6ba1fcccedc6bcf3f9a8"} Nov 02 04:11:12.485284 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:12.485277 4868 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" event={"ID":"c7b37551-2bff-4657-a50b-4894c61dc3f4","Type":"ContainerStarted","Data":"8e20e2918f7b00dac0c846bff597f3e76af72f6ad2bc434ae3f6fcdf27f66c10"} Nov 02 04:11:12.485455 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:12.485305 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:12.599549 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:12.599512 4868 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" podStartSLOduration=1.5994996000000001 podStartE2EDuration="1.5994996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 04:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 04:11:12.597929541 +0000 UTC m=+5014.593630484" watchObservedRunningTime="2026-11-02 04:11:12.5994996 +0000 UTC m=+5014.595200592" Nov 02 04:11:16.338679 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:16.338637 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6spg5_e07d60c1-9802-4fa0-9a5b-85ec83152e37/manager/0.log" Nov 02 04:11:16.634222 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:16.634191 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dbf54fd5c-ttkx8_f5e7e1a3-15ef-485e-a84e-a257d62e8f14/manager/0.log" Nov 02 04:11:18.500861 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:18.500831 4868 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbmzv/perf-node-gather-daemonset-jl9b9" Nov 02 04:11:19.571587 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:19.571556 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-sgsh7_201c03d0-ed5b-4ccb-b4c2-534fe47511c4/cluster-monitoring-operator/0.log" Nov 02 04:11:21.798054 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:21.798021 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lqm9w_e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9/discovery/0.log" Nov 02 04:11:22.076804 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:22.076728 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2gs9s_fee48a3f-91ff-420f-bf42-8acc0d7877a9/node-exporter/0.log" Nov 02 04:11:22.126021 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:22.125994 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66599bc4bb-z7fp2_8e6b08ff-10f3-4655-b67b-18931ee4162a/kube-auth-proxy/0.log" Nov 02 04:11:22.491570 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:22.491540 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2gs9s_fee48a3f-91ff-420f-bf42-8acc0d7877a9/kube-rbac-proxy/0.log" Nov 02 04:11:22.903235 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:22.903210 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2gs9s_fee48a3f-91ff-420f-bf42-8acc0d7877a9/init-textfile/0.log" Nov 02 04:11:23.184004 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:23.183939 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-648db889-4lftz_d25eaef0-b0ac-4326-925b-aaef13840661/router/0.log" Nov 02 04:11:29.193526 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:29.193499 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-84qbj_cce70044-e2d2-40aa-a55b-f1ba8f1cc976/kuadrant-console-plugin/0.log" Nov 02 04:11:33.445312 ip-10-0-137-74 kubenswrapper[4868]: E1102 04:11:33.445278 4868 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5jsrl" containerName="extractor" Nov 02 04:11:35.372077 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:35.372050 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lqm9w_e2eaf6cc-9d41-43cb-98e2-1a5d910fa4f9/discovery/0.log" Nov 02 04:11:35.687821 ip-10-0-137-74 kubenswrapper[4868]: I1102 04:11:35.687747 4868 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66599bc4bb-z7fp2_8e6b08ff-10f3-4655-b67b-18931ee4162a/kube-auth-proxy/0.log"